search

fjall-rs / fjall

LSM-based embeddable key-value storage engine written in safe Rust
https://fjall-rs.github.io/
Apache License 2.0
323 stars 13 forks source link

unexpected warning ： shard.rs:106: Invalid batch: found batch start inside batch #53

Closed i18nsite closed 3 months ago

i18nsite commented 3 months ago

I'm writing a command line application. When my program exits, I use keyspace.persist(FlushMode::SyncAll)?; I not close the database (is there a shutdown function?)

The following warn occasionally appears at 2 times start my cli. ( video https://www.loom.com/share/c1c1a71d06fb495bad85bacda73b219e )

Is there a way to block this warning?

  WARN fjall::journal::shard: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.3/src/journal/shard.rs:106: Invalid batch: found batch start inside batch
  WARN fjall::journal::shard: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.3/src/journal/shard.rs:70: Truncating shard to 88
  WARN fjall::journal::shard: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.3/src/journal/shard.rs:213: Invalid batch: missing terminator, but last batch, so probably incomplete, discarding to keep atomicity
  WARN fjall::journal::shard: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.3/src/journal/shard.rs:70: Truncating shard to 88
  WARN fjall::journal::shard: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.3/src/journal/shard.rs:106: Invalid batch: found batch start inside batch
  WARN fjall::journal::shard: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.3/src/journal/shard.rs:70: Truncating shard to 0
  WARN fjall::journal::shard: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.3/src/journal/shard.rs:213: Invalid batch: missing terminator, but last batch, so probably incomplete, discarding to keep atomicity
  WARN fjall::journal::shard: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.3/src/journal/shard.rs:70: Truncating shard to 0
  WARN fjall::journal::shard: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.3/src/journal/shard.rs:106: Invalid batch: found batch start inside batch
  WARN fjall::journal::shard: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.3/src/journal/shard.rs:70: Truncating shard to 0
  WARN fjall::journal::shard: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.3/src/journal/shard.rs:213: Invalid batch: missing terminator, but last batch, so probably incomplete, discarding to keep atomicity
  WARN fjall::journal::shard: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.3/src/journal/shard.rs:70: Truncating shard to 0
  WARN fjall::journal::shard: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.3/src/journal/shard.rs:106: Invalid batch: found batch start inside batch
  WARN fjall::journal::shard: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.3/src/journal/shard.rs:70: Truncating shard to 0
  WARN fjall::journal::shard: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.3/src/journal/shard.rs:213: Invalid batch: missing terminator, but last batch, so probably incomplete, discarding to keep atomicity
  WARN fjall::journal::shard: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.3/src/journal/shard.rs:70: Truncating shard to 0
marvin-j97 commented 3 months ago

When my program exits, I use keyspace.persist(FlushMode::SyncAll)?;

On Drop, flush is called automatically https://github.com/fjall-rs/fjall/blob/main/src/keyspace.rs#L80

I not close the database (is there a shutdown function?)

Dropping is the shutdown function. You can check trace logs to see if the keyspace is properly dropping in your application.

Is there a way to block this warning?

I may need to adjust the verbosity of some of the logging. Depending on the logger you use, you may be able to set a filter though.

marvin-j97 commented 3 months ago

I have adjusted the logging levels to be debug and trace in the next version

i18nsite commented 3 months ago

more log

❯ ./i18/test.sh
+ '[' -z '' ']'
++ realpath /Users/z/i18n/rust/i18
+ PROJECT=/Users/z/i18n/rust/i18/tests/yml
+ direnv exec . cargo run -- -d /Users/z/i18n/rust/i18/tests/yml
direnv: loading ~/i18n/rust/i18/.envrc
direnv: loading ~/i18n/rust/.envrc
   Compiling i18 v0.1.203 (/Users/z/i18n/rust/i18)
    Finished `dev` profile [optimized + debuginfo] target(s) in 1.57s
     Running `/tmp/rust/target/debug/i18 -d /Users/z/i18n/rust/i18/tests/yml`
  INFO fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:271: Opening keyspace at "/Users/z/i18n/rust/i18/tests/yml/.i18n/cache"
  INFO fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:573: Creating keyspace at "/Users/z/i18n/rust/i18/tests/yml/.i18n/cache"
  INFO fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:299: Spawning 11 compaction threads
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::partition: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/partition/mod.rs:121: Creating partition md
 DEBUG lsm_tree::tree: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lsm-tree-1.0.4/src/tree.rs:59: Opening LSM-tree at "/Users/z/i18n/rust/i18/tests/yml/.i18n/cache/partitions/md"
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::partition: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/partition/mod.rs:121: Creating partition yml
 DEBUG lsm_tree::tree: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lsm-tree-1.0.4/src/tree.rs:59: Opening LSM-tree at "/Users/z/i18n/rust/i18/tests/yml/.i18n/cache/partitions/yml"
README.md ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── 0% ⠁ 00:00:00 ETA 0s DEBUG reqwest::connect: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.12.4/src/connect.rs:497: starting new connection: http://38.242.220.222:8850/
README.md ==================================================================================================================================================== 100%   00:00:00 ETA 0s DEBUG fjall::flush::worker: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/flush/worker.rs:113: flush worker: write locking flush manager
 DEBUG fjall::flush::worker: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/flush/worker.rs:121: flush worker: No tasks collected
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:660: fsync thread: exiting because tree is dropping

✅ i18n.site translate
❯ ./i18/test.sh
+ '[' -z '' ']'
++ realpath /Users/z/i18n/rust/i18
+ PROJECT=/Users/z/i18n/rust/i18/tests/yml
+ direnv exec . cargo run -- -d /Users/z/i18n/rust/i18/tests/yml
direnv: loading ~/i18n/rust/i18/.envrc
direnv: loading ~/i18n/rust/.envrc
   Compiling i18 v0.1.203 (/Users/z/i18n/rust/i18)
    Finished `dev` profile [optimized + debuginfo] target(s) in 1.63s
     Running `/tmp/rust/target/debug/i18 -d /Users/z/i18n/rust/i18/tests/yml`
  INFO fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:271: Opening keyspace at "/Users/z/i18n/rust/i18/tests/yml/.i18n/cache"
  INFO fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:516: Recovering keyspace at "/Users/z/i18n/rust/i18/tests/yml/.i18n/cache"
  INFO fjall::journal: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/journal/mod.rs:64: Recovering journal from "/Users/z/i18n/rust/i18/tests/yml/.i18n/cache/journals/0"
 DEBUG fjall::journal::shard: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/journal/shard.rs:106: Invalid batch: found batch start inside batch
 DEBUG fjall::journal::shard: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/journal/shard.rs:213: Invalid batch: missing terminator, but last batch, so probably incomplete, discarding to keep atomicity
 DEBUG fjall::journal::shard: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/journal/shard.rs:106: Invalid batch: found batch start inside batch
 DEBUG fjall::journal::shard: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/journal/shard.rs:213: Invalid batch: missing terminator, but last batch, so probably incomplete, discarding to keep atomicity
 DEBUG fjall::journal::shard: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/journal/shard.rs:106: Invalid batch: found batch start inside batch
 DEBUG fjall::journal::shard: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/journal/shard.rs:213: Invalid batch: missing terminator, but last batch, so probably incomplete, discarding to keep atomicity
 DEBUG fjall::journal::shard: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/journal/shard.rs:106: Invalid batch: found batch start inside batch
 DEBUG fjall::journal::shard: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/journal/shard.rs:213: Invalid batch: missing terminator, but last batch, so probably incomplete, discarding to keep atomicity
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:528: Recovered active journal at "/Users/z/i18n/rust/i18/tests/yml/.i18n/cache/journals/0"
 DEBUG lsm_tree::tree: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lsm-tree-1.0.4/src/tree.rs:59: Opening LSM-tree at "/Users/z/i18n/rust/i18/tests/yml/.i18n/cache/partitions/md"
  INFO lsm_tree::tree: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lsm-tree-1.0.4/src/tree.rs:798: Recovering LSM-tree at "/Users/z/i18n/rust/i18/tests/yml/.i18n/cache/partitions/md"
 DEBUG lsm_tree::tree: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lsm-tree-1.0.4/src/tree.rs:936: Recovering disk segments from "/Users/z/i18n/rust/i18/tests/yml/.i18n/cache/partitions/md"
 DEBUG lsm_tree::tree: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lsm-tree-1.0.4/src/tree.rs:997: Recovered 0 segments
 DEBUG fjall::recovery: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/recovery.rs:101: Keyspace seqno is now 2
 DEBUG lsm_tree::tree: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lsm-tree-1.0.4/src/tree.rs:59: Opening LSM-tree at "/Users/z/i18n/rust/i18/tests/yml/.i18n/cache/partitions/yml"
  INFO lsm_tree::tree: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lsm-tree-1.0.4/src/tree.rs:798: Recovering LSM-tree at "/Users/z/i18n/rust/i18/tests/yml/.i18n/cache/partitions/yml"
 DEBUG lsm_tree::tree: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lsm-tree-1.0.4/src/tree.rs:936: Recovering disk segments from "/Users/z/i18n/rust/i18/tests/yml/.i18n/cache/partitions/yml"
 DEBUG lsm_tree::tree: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lsm-tree-1.0.4/src/tree.rs:997: Recovered 0 segments
 DEBUG fjall::recovery: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/recovery.rs:101: Keyspace seqno is now 2
  INFO fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:299: Spawning 11 compaction threads
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::flush::worker: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/flush/worker.rs:113: flush worker: write locking flush manager
 DEBUG fjall::flush::worker: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/flush/worker.rs:121: flush worker: No tasks collected
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping

✅ i18n.site translate
❯ ./i18/test.sh
+ '[' -z '' ']'
++ realpath /Users/z/i18n/rust/i18
+ PROJECT=/Users/z/i18n/rust/i18/tests/yml
+ direnv exec . cargo run -- -d /Users/z/i18n/rust/i18/tests/yml
direnv: loading ~/i18n/rust/i18/.envrc
direnv: loading ~/i18n/rust/.envrc
    Finished `dev` profile [optimized + debuginfo] target(s) in 0.27s
     Running `/tmp/rust/target/debug/i18 -d /Users/z/i18n/rust/i18/tests/yml`
  INFO fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:271: Opening keyspace at "/Users/z/i18n/rust/i18/tests/yml/.i18n/cache"
  INFO fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:516: Recovering keyspace at "/Users/z/i18n/rust/i18/tests/yml/.i18n/cache"
  INFO fjall::journal: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/journal/mod.rs:64: Recovering journal from "/Users/z/i18n/rust/i18/tests/yml/.i18n/cache/journals/0"
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:528: Recovered active journal at "/Users/z/i18n/rust/i18/tests/yml/.i18n/cache/journals/0"
 DEBUG lsm_tree::tree: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lsm-tree-1.0.4/src/tree.rs:59: Opening LSM-tree at "/Users/z/i18n/rust/i18/tests/yml/.i18n/cache/partitions/md"
  INFO lsm_tree::tree: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lsm-tree-1.0.4/src/tree.rs:798: Recovering LSM-tree at "/Users/z/i18n/rust/i18/tests/yml/.i18n/cache/partitions/md"
 DEBUG lsm_tree::tree: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lsm-tree-1.0.4/src/tree.rs:936: Recovering disk segments from "/Users/z/i18n/rust/i18/tests/yml/.i18n/cache/partitions/md"
 DEBUG lsm_tree::tree: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lsm-tree-1.0.4/src/tree.rs:997: Recovered 0 segments
 DEBUG fjall::recovery: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/recovery.rs:101: Keyspace seqno is now 2
 DEBUG lsm_tree::tree: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lsm-tree-1.0.4/src/tree.rs:59: Opening LSM-tree at "/Users/z/i18n/rust/i18/tests/yml/.i18n/cache/partitions/yml"
  INFO lsm_tree::tree: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lsm-tree-1.0.4/src/tree.rs:798: Recovering LSM-tree at "/Users/z/i18n/rust/i18/tests/yml/.i18n/cache/partitions/yml"
 DEBUG lsm_tree::tree: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lsm-tree-1.0.4/src/tree.rs:936: Recovering disk segments from "/Users/z/i18n/rust/i18/tests/yml/.i18n/cache/partitions/yml"
 DEBUG lsm_tree::tree: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lsm-tree-1.0.4/src/tree.rs:997: Recovered 0 segments
 DEBUG fjall::recovery: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/recovery.rs:101: Keyspace seqno is now 2
  INFO fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:299: Spawning 11 compaction threads
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:673: compaction: waiting for work
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::flush::worker: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/flush/worker.rs:113: flush worker: write locking flush manager
 DEBUG fjall::flush::worker: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/flush/worker.rs:121: flush worker: No tasks collected
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
 DEBUG fjall::keyspace: /Users/z/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fjall-1.0.4/src/keyspace.rs:679: compaction thread: exiting because tree is dropping
marvin-j97 commented 3 months ago

@i18n-now Looks good to me, do you expect different logs? The missing terminator, but last batch logs are to be expected, I think with V2 I will be able to get rid of them... but they don't necessarily indicate data loss.

i18nsite commented 3 months ago

I see , I thought it was a problem with my usage that caused the error.

If the data is automatically written when dropped, why is there still a warning?

I would expect a way to explicitly avoid this prompt, as the data being possibility correct also means there is a possibility of loss.

Losing data is quite terrifying.

Additionally, I am using fjall as a cache in my translation tool (which is still under development): https://github.com/search?q=repo%3Ai18n-site%2Frust%20fjall&type=code

image

The project is currently under development, and Fjall is very useful ^_^.

I hope there can be a performance monitoring comparison that can monitor the performance of each version. Something similar to https://github.com/benchmark-action/github-action-benchmark

image

marvin-j97 commented 3 months ago

I see , I thought it was a problem with my usage that caused the error.

If the data is automatically written when dropped, why is there still a warning?

That's a bit of an oversight on my part, it's not critical, but it would be a breaking change to "fix" it. So it's gonna be a data format V2 change.

I would expect a way to explicitly avoid this prompt, as the data being possibility correct also means there is a possibility of loss.

Losing data is quite terrifying.

Me too honestly, it's not 100% transparent right now, but it's fixable.

I hope there can be a performance monitoring comparison that can monitor the performance of each version. Something similar to https://github.com/benchmark-action/github-action-benchmark

I planned on adding something like this, but it's a bit of effort to setup. I didn't know about that GH action, thanks for sharing!