Some problems with database

reyamir commented 1 month ago

Describe the bug
I've a client with database like this

// Create nostr database connection
let db_path = home_dir.join(&"Lume/database");

#[cfg(target_family = "unix")]
let database = NdbDatabase::open(db_path.to_str().unwrap());

// #[cfg(target_os = "windows")]
// let database = RocksDatabase::open(db_path.to_str().unwrap()).await;

// Create nostr connection
let client = match database {
  Ok(db) => ClientBuilder::default().database(db).build(),
  Err(_) => ClientBuilder::default().build(),
};

Then I make a query directly to database like this

let filter = Filter::new().pubkey(public_key).kinds(vec![
  Kind::TextNote,
  Kind::Repost,
  Kind::Reaction,
  Kind::ZapReceipt,
]).limit(20);

match client.database().query(vec![filter], Order::default()).await
{
  Ok(events) => Ok(events.into_iter().map(|ev| ev.as_json()).collect()),
  Err(err) => Err(err.to_string()),
}

It will return empty events, but if I use RocksDB or SQLite, it will return exactly events that match my filter.

To Reproduce

Create a client which use nostr-db
Create a query directly to database

Expected behavior
Return exactly events that match the filter like RocksDB or SQLite

Build environment

Library: nostr-sdk
Language: Rust
Language version:
Tag/commit:
OS+version: macOs

reyamir commented 1 month ago

Oh, I'm also notify if I use database rather than SQLite, I cannot receive new notification too.

let client = &state.client;

// Create a subscription for notification
let notification_id = SubscriptionId::new("notification");
let filter = Filter::new()
  .pubkey(public_key)
  .kinds(vec![
    Kind::TextNote,
    Kind::Repost,
    Kind::Reaction,
    Kind::ZapReceipt,
  ])
  .since(Timestamp::now());

// Subscribe
client
  .subscribe_with_id(notification_id.clone(), vec![filter], None)
  .await;

// Handle notifications
let _ = client
  .handle_notifications(|notification| async {
    if let RelayPoolNotification::Event {
      subscription_id,
      event,
      ..
    } = notification
    {
      if subscription_id == notification_id {
        println!("new notification: {}", event.as_json());
      }
    }
    Ok(false)
  })
  .await;

reyamir commented 1 month ago

And if I use SQLite as database, the query order is not work as expect too.

let events = client.database().query(vec![filter], Order::default()) <- events will return as random order

yukibtc commented 1 month ago

Oh, I'm also notify if I use database rather than SQLite, I cannot receive new notification too.

let client = &state.client;

// Create a subscription for notification
let notification_id = SubscriptionId::new("notification");
let filter = Filter::new()
  .pubkey(public_key)
  .kinds(vec![
    Kind::TextNote,
    Kind::Repost,
    Kind::Reaction,
    Kind::ZapReceipt,
  ])
  .since(Timestamp::now());

// Subscribe
client
  .subscribe_with_id(notification_id.clone(), vec![filter], None)
  .await;

// Handle notifications
let _ = client
  .handle_notifications(|notification| async {
    if let RelayPoolNotification::Event {
      subscription_id,
      event,
      ..
    } = notification
    {
      if subscription_id == notification_id {
        println!("new notification: {}", event.as_json());
      }
    }
    Ok(false)
  })
  .await;

This one is not a bug. RelayPoolNotification::Event variant is sent only the first time the event is seen. When it's saved, notification is no longer sent. You can use the RelayPoolNotification::Message variant instead (it's always sent).

yukibtc commented 1 month ago

And if I use SQLite as database, the query order is not work as expect too.
let events = client.database().query(vec![filter], Order::default()) <- events will return as random order

Thanks, fixed at 6c41bd6709735e63c600f015728d51649188a553

yukibtc commented 1 month ago

Regarding nostrdb, I checked the code on the rust-nostr side and seems correct. It just call nostrdb methods. If I'm not doing something wrong, the issue could be related to nostrdb.

https://github.com/rust-nostr/nostr/blob/master/crates%2Fnostr-ndb%2Fsrc%2Flib.rs

@jb55

jb55 commented 4 weeks ago

On Wed, Jun 05, 2024 at 05:48:19AM GMT, Yuki Kishimoto wrote:

Regarding nostrdb, I checked the code on the rust-nostr side and seems correct. It just call nostrdb methods. If I'm not doing something wrong, the issue could be related to nostrdb.

https://github.com/rust-nostr/nostr/blob/master/crates%2Fnostr-ndb%2Fsrc%2Flib.rs

@jb55

I've done these queries locally on nostrdb directly (via ndb) and I can't seem to reproduce it. may need a minimal reproducible rust example to test with.

yukibtc commented 4 weeks ago

@jb55, here is the repo with the example to reproduce it: https://github.com/yukibtc/rust-nostr-db-issue.

It execute a reconciliation with damus relay of all notes authored by me and query {"authors":["68d81165918100b7da43fc28f7d1fc12554466e1115886b9e7bb326f65ec4272"],"kinds":[1],"limit":5} from local database.

The SQLite return these events (which seems right, according to other clients):

{"id":"296dcaf1d8f9a28e4e9f8f1cca63acff215447765248c58053a96252fb82d19b","pubkey":"68d81165918100b7da43fc28f7d1fc12554466e1115886b9e7bb326f65ec4272","created_at":1716914958,"kind":1,"tags":[["e","fde4199cf10077baa9f08d146f20d5a7a1e62696749f9f2d69459e134c0480e3","","root"],["p","99bb5591c9116600f845107d31f9b59e2f7c7e09a1ff802e84f1d43da557ca64"],["p","da18e9860040f3bf493876fc16b1a912ae5a6f6fa8d5159c3de2b8233a0d9851"],["p","e4336cd525df79fa4d3af364fd9600d4b10dce4215aa4c33ed77ea0842344b10"],["p","89ef92b9ebe6dc1e4ea398f6477f227e95429627b0a33dc89b640e137b256be5"],["p","68d81165918100b7da43fc28f7d1fc12554466e1115886b9e7bb326f65ec4272"]],"content":"Publishing `0.13.0-alpha.2` that fixes some issues related to async changes.\n\nPython devs that receive \"no running event loop\" have to add `uniffi_set_event_loop(asyncio.get_running_loop())` line to the code.","sig":"c1d05d2339d10753c89535168c4d79db1df92ba3efb713957314775d53577816342b92ad57c4dbe4b7dc4b201e42246fa6aa7a18546158419971cec699c78d2e"}
{"id":"fde4199cf10077baa9f08d146f20d5a7a1e62696749f9f2d69459e134c0480e3","pubkey":"68d81165918100b7da43fc28f7d1fc12554466e1115886b9e7bb326f65ec4272","created_at":1716903182,"kind":1,"tags":[["t","rustnostr"],["p","99bb5591c9116600f845107d31f9b59e2f7c7e09a1ff802e84f1d43da557ca64","","mention"],["p","da18e9860040f3bf493876fc16b1a912ae5a6f6fa8d5159c3de2b8233a0d9851","","mention"],["p","e4336cd525df79fa4d3af364fd9600d4b10dce4215aa4c33ed77ea0842344b10","","mention"],["p","89ef92b9ebe6dc1e4ea398f6477f227e95429627b0a33dc89b640e137b256be5","","mention"]],"content":"For all devs using the Python, Kotlin or Swift rust-nostr libraries: I've published an alpha version of nostr-sdk with async/future support. Please give it a try and give me a feedback.\n\nAll rust-nostr libraries (except for protocol implementation) are async so make sense to convert bindings to async instead of keeping them blocking, but I don't know how common async programming is in those langs.\n\nThe alpha versions are:\n* Python: 0.13.0a1.dev0\n* Kotlin: 0.13.0-alpha.1\n* Swift: 0.13.0-alpha.1\n\nHere you can find some async python examples: https://github.com/rust-nostr/nostr/tree/async-ffi/bindings%2Fnostr-sdk-ffi%2Fbindings-python%2Fexamples\n \nhttps://github.com/rust-nostr/nostr/pull/379\n\nnostr:npub1nxa4tywfz9nqp7z9zp7nr7d4nchhclsf58lcqt5y782rmf2hefjquaa6q8 nostr:npub1mgvwnpsqgrem7jfcwm7pdvdfz2h95mm04r23t8pau2uzxwsdnpgs0gpdjc nostr:npub1useke4f9maul5nf67dj0m9sq6jcsmnjzzk4ycvldwl4qss35fvgqjdk5ks nostr:npub138he9w0tumwpun4rnrmywlez06259938kz3nmjymvs8px7e9d0js8lrdr2\n\n#rustnostr","sig":"383feff9a4fe38ad62b7719e3b80b99980b6f952b820026a708cd04e3c719099c7e841506aab7ea598991842365c603095501a4df1a3067ce96e2148c9837338"}
{"id":"1c8a456aa61262673186adadb9758e4c8bf2642224f7e0f1ead80275b6edc6b2","pubkey":"68d81165918100b7da43fc28f7d1fc12554466e1115886b9e7bb326f65ec4272","created_at":1716833943,"kind":1,"tags":[["e","06269da7dbf8d9b6b1b14a9017c6a05e05c999b98e74c8e0aea1b6ba7d42911a","wss://nostr.mom/","root"],["e","f12b0d3a0bc9717084e5e7f35a6d55c1ccd74a088acfcc82bcc5d631f1ba8577","wss://nostr.oxtr.dev/","reply"],["p","38e02d76b4299acea0c847217b602bce12a9d3cea3cabdfd616c58324cff73d3","","mention"]],"content":"I'm compiling the wheels only for some python versions and arch, so maybe it's not available the one for your setup. Here you can check the support matrix: https://rust-nostr.org/nostr-sdk/02-installation.html#support-matrix\n\nIf miss your one, you can compile the wheel by yourself: \n* Clone rust-nostr repo\n* `cd bindings/nostr-sdk-ffi` (or `cd bindings/nostr-ffi`)\n* Install `rust` and `just`\n* Run `just python`\n\nThe wheel will be located in `bindings-python/dist`","sig":"50ebf998ac7175e3c883bbba488928db19852af44ba088632c9d35b8acc8a36e3f6a312216772a8cad20d8cb484b18141bad44c9f84e612e2a3e2af8e985d2bc"}
{"id":"8ba5ff1e7fa6313bf21edbff9fc930b1b85fb5cbb9333d6ac1ea61d4116c29d2","pubkey":"68d81165918100b7da43fc28f7d1fc12554466e1115886b9e7bb326f65ec4272","created_at":1716830422,"kind":1,"tags":[["e","06269da7dbf8d9b6b1b14a9017c6a05e05c999b98e74c8e0aea1b6ba7d42911a","","root"],["e","5c411e3f150ee3438dfc167d0e45bf062aa00164d123357ca4386f4eab81ed7b","","reply"],["p","a008def15796fba9a0d6fab04e8fd57089285d9fd505da5a83fe8aad57a3564d"],["p","68d81165918100b7da43fc28f7d1fc12554466e1115886b9e7bb326f65ec4272"],["p","38e02d76b4299acea0c847217b602bce12a9d3cea3cabdfd616c58324cff73d3"]],"content":"Hey, I not tested on Alpine yet. Have you any issue?","sig":"fac2770f5c294b4b99183746f905ca63ba1be6b6a6f87435a9a76791d0afbb7855f3d2be459ee9437b41093c8dff493fa9c0b63d1016ff9f6ec5f1f53418996d"}
{"id":"06269da7dbf8d9b6b1b14a9017c6a05e05c999b98e74c8e0aea1b6ba7d42911a","pubkey":"68d81165918100b7da43fc28f7d1fc12554466e1115886b9e7bb326f65ec4272","created_at":1715941473,"kind":1,"tags":[["t","rustnostr"],["t","nostr"],["p","a008def15796fba9a0d6fab04e8fd57089285d9fd505da5a83fe8aad57a3564d","","mention"]],"content":"## rust-nostr release is out! 🦀\n\n### Versions\n\nRust: v0.31\nJavaScript: v0.14\nPython, Kotlin and Swift: v0.12\n\n### Summary\n\nReworked `Tag`, added `TagStandard` enum, simplified the way to subscribe and/or reconcile to subset of relays (respectively, `client.subscribe_to` and `client.reconcile_with`), added blacklist support to mute public keys or event IDs, removed zap split from `client.zap` method, many improvements and more!\n\nFull changelog: https://rust-nostr.org/changelog\n\n### Contributors\n\nThanks to nostr:npub15qydau2hjma6ngxkl2cyar74wzyjshvl65za5k5rl69264ar2exs5cyejr for contributing!\n\n### Links\n\nhttps://rust-nostr.org\nhttps://rust-nostr.org/donate\n\n#rustnostr #nostr","sig":"6580965b8f1b84b0d4a823ed913d0809f8ef68b3a6f640b181dea94a91c28df044442d42a19e819c7c13c1929a3318d5ad0e627240b98270472ea5214f8dcd44"}

The ndb return these events:

{"id":"fde4199cf10077baa9f08d146f20d5a7a1e62696749f9f2d69459e134c0480e3","pubkey":"68d81165918100b7da43fc28f7d1fc12554466e1115886b9e7bb326f65ec4272","created_at":1716903182,"kind":1,"tags":[["t","rustnostr"],["p","99bb5591c9116600f845107d31f9b59e2f7c7e09a1ff802e84f1d43da557ca64","","mention"],["p","da18e9860040f3bf493876fc16b1a912ae5a6f6fa8d5159c3de2b8233a0d9851","","mention"],["p","e4336cd525df79fa4d3af364fd9600d4b10dce4215aa4c33ed77ea0842344b10","","mention"],["p","89ef92b9ebe6dc1e4ea398f6477f227e95429627b0a33dc89b640e137b256be5","","mention"]],"content":"For all devs using the Python, Kotlin or Swift rust-nostr libraries: I've published an alpha version of nostr-sdk with async/future support. Please give it a try and give me a feedback.\n\nAll rust-nostr libraries (except for protocol implementation) are async so make sense to convert bindings to async instead of keeping them blocking, but I don't know how common async programming is in those langs.\n\nThe alpha versions are:\n* Python: 0.13.0a1.dev0\n* Kotlin: 0.13.0-alpha.1\n* Swift: 0.13.0-alpha.1\n\nHere you can find some async python examples: https://github.com/rust-nostr/nostr/tree/async-ffi/bindings%2Fnostr-sdk-ffi%2Fbindings-python%2Fexamples\n \nhttps://github.com/rust-nostr/nostr/pull/379\n\nnostr:npub1nxa4tywfz9nqp7z9zp7nr7d4nchhclsf58lcqt5y782rmf2hefjquaa6q8 nostr:npub1mgvwnpsqgrem7jfcwm7pdvdfz2h95mm04r23t8pau2uzxwsdnpgs0gpdjc nostr:npub1useke4f9maul5nf67dj0m9sq6jcsmnjzzk4ycvldwl4qss35fvgqjdk5ks nostr:npub138he9w0tumwpun4rnrmywlez06259938kz3nmjymvs8px7e9d0js8lrdr2\n\n#rustnostr","sig":"383feff9a4fe38ad62b7719e3b80b99980b6f952b820026a708cd04e3c719099c7e841506aab7ea598991842365c603095501a4df1a3067ce96e2148c9837338"}
{"id":"df5ec668434d8fe3216243161c0a9754e84d010735ebf3bfaad744cc42ed36cd","pubkey":"68d81165918100b7da43fc28f7d1fc12554466e1115886b9e7bb326f65ec4272","created_at":1715365769,"kind":1,"tags":[["e","572c0f8f386443c5761eeb750bff15f3f2a34653e067d2904d87deec86a8cacb","","root"],["e","49974751f9df9493f3d062f0a32b5991955b983ad807663e39ccce31abea1e53"],["e","bffa827c9db46ef6660eb1311642756dc8703776e76651133c527c4d9fd164d3","","reply"],["p","e1ff3bfdd4e40315959b08b4fcc8245eaa514637e1d4ec2ae166b743341be1af"],["p","68d81165918100b7da43fc28f7d1fc12554466e1115886b9e7bb326f65ec4272"],["p","50d94fc2d8580c682b071a542f8b1e31a200b0508bab95a33bef0855df281d63"],["r","https://github.com/hoytech/negentropy"]],"content":"There isn't a NIP yet, but you can learn more here: https://github.com/hoytech/negentropy","sig":"6da0bdac2a4e4f3879403687cd121de724784269359ab6792be0c7113221ff06e069c022928b5e498bf5478ed6e55fda274b0786a64fb2d7d79ee2b2c70fe0de"}
{"id":"fc3eb4ad9359733dc55b5a3c5835c29eb5f47fb11ab7f536e8040191f46f6811","pubkey":"68d81165918100b7da43fc28f7d1fc12554466e1115886b9e7bb326f65ec4272","created_at":1715150882,"kind":1,"tags":[["e","a6b02ce0eaa96dea65ea2957e267b7caaf350d375bf21d2e37e1dbf80a3a3971","","root"],["p","fa984bd7dbb282f07e16e7ae87b26a2a7b9b90b7246a44771f0cf5ae58018f52"],["p","99bb5591c9116600f845107d31f9b59e2f7c7e09a1ff802e84f1d43da557ca64"],["p","68d81165918100b7da43fc28f7d1fc12554466e1115886b9e7bb326f65ec4272"],["p","da18e9860040f3bf493876fc16b1a912ae5a6f6fa8d5159c3de2b8233a0d9851"],["r","https://github.com/rust-nostr/nostr/blob/master/bindings%2Fnostr-sdk-ffi%2Fbindings-python%2Fexamples%2Fbot.py"]],"content":"`client.get_events_of` it's basically an auto-closing subscription: subscribe, listen until EOSE or when timeout is reached, close subscription and return events.\n\nTo subscribe \"forever\", just use the `client. subscribe` method. All the events will be sent to a notification channel. You can find an example here: \n\nhttps://github.com/rust-nostr/nostr/blob/master/bindings%2Fnostr-sdk-ffi%2Fbindings-python%2Fexamples%2Fbot.py","sig":"2fe7ddaf8cd65bac25e49a3bbc02a257cd1089978c1254904865d2869beb0cc4661e3377679810d4e0cd9b826e6aa6b8d812a56e58e0eb258849bb93a1c03906"}
{"id":"f8c1d62434906d77ee9ddb7efb0bf6b7bb0e958a49796686f07fe05c8b5f827f","pubkey":"68d81165918100b7da43fc28f7d1fc12554466e1115886b9e7bb326f65ec4272","created_at":1707229988,"kind":1,"tags":[["e","a83add9517a5afe8d3ab57fababa4fe6c38b1c28d4526d9f35c9080e17be1d97","","root"],["p","126103bfddc8df256b6e0abfd7f3797c80dcc4ea88f7c2f87dd4104220b4d65f"],["r","v0.27"],["r","Cargo.toml"],["r","v0.28"],["r","client.database"]],"content":"The v0.27 have support to negentropy but have some issues. I fixed all the issues in master branch (you can specify a git repo and commit in the Cargo.toml or wait for v0.28)\n\nTo use negentropy you need a persistent DB. Configured that with the ClientBuilder, just call \"client.reconcile\". All the reconciled event will be saved in the db: use client.database().query(filters) to query what you need.","sig":"d561dc2bce48869756000d8d53ac213b6061e1e34137908f5db5a52d2a7e08a58286542790da0088cc2e634f7b075dda4194acc085aa900142a4c9dbb37bd7d2"}
{"id":"d88e436d92257146f44c8cc7b0b7cf6412ee259b11f615f351085ac881ed2e72","pubkey":"68d81165918100b7da43fc28f7d1fc12554466e1115886b9e7bb326f65ec4272","created_at":1702856454,"kind":1,"tags":[["e","d1296b09f567eee81d57e003185f97dbe9f1d4b64ea0e141f6330d84e1a5e5dd","","root"],["e","189bf17113ca6acbc605d6e6dcf2b51d30a07baf03f793bada70eb3bd9b638a5"],["e","7c87e2076403765a1358abac8da5335bb566abac566bbef819937e3421ee16c4","","reply"],["p","68d81165918100b7da43fc28f7d1fc12554466e1115886b9e7bb326f65ec4272"],["p","f0293508f3eb9e6fe99c2fd8ba69ff446216872a2d9f67979bfa4db8b3155806"],["p","32e1827635450ebb3c5a7d12c1f8e7b2b514439ac10a67eef3d9fd9c5c68e245"],["p","8fb9450003a599bb1b34f03fadb9b137f6c0e5a850ba205964bee4732ccce549"]],"content":"It's possible to integrate any database, also ones that not support arbitrary queries, by using the `DatabaseIndexes`.\n\nAll the current backends available for nostr-sdk (SQLite, indexeddb, RocksDB, ...) rely on it, also SQLite that has the ability to perform complex queries.\n\nThe flow under the hood is: build in-memory indexes (700_000 events are indexed in ~1.5 sec, not faster due to param/replaceable and deletion events checks), query indexes by filters and use the output (event IDs) to get full events from persistent DB.\n","sig":"5ab100d7650168e03f2ffca086fddcbbbf41e5c15b0bc635d89107995090fd237b16944e50f4dcf8c12a60141258060c3b4ae6b57a4b0bfbb8503ee20be384a1"}

nostrdb seems to have stored exactly the same events of SQLite, so probably it's just an issue with query. For example, if I query nostrdb with {"authors":["68d81165918100b7da43fc28f7d1fc12554466e1115886b9e7bb326f65ec4272"],"kinds":[1],"limit":5} filter, it not return the 8ba5ff1e7fa6313bf21edbff9fc930b1b85fb5cbb9333d6ac1ea61d4116c29d2 event. But if I query it by explicit asking for that ID, it return it.

jb55 commented 3 weeks ago

On Thu, Jun 06, 2024 at 10:25:29AM GMT, Yuki Kishimoto wrote:

@jb55, here is the repo with the example to reproduce it: https://github.com/yukibtc/rust-nostr-db-issue.

I'll take a look at this soon, but at first glance I don't see any code that waits for the notes to finish processing in the database. You would need a subscription for this. nostrdb processess things asyncronously in a threadpool.

so if you do a query before ingest is complete, you likely wouldn't have the complete resultset right away?

yukibtc commented 4 days ago

So if you do a query before ingest is complete, you likely wouldn't have the complete resultset right away?

Sorry for late reply. Yes, but I noticed that if I query directly the ID the event is returned. If I query by author and kind event is not returned. So I guess the ingestion is completed.

Example

This works:

{"ids":["8ba5ff1e7fa6313bf21edbff9fc930b1b85fb5cbb9333d6ac1ea61d4116c29d2"]}

This NOT works (well):

{"authors":["68d81165918100b7da43fc28f7d1fc12554466e1115886b9e7bb326f65ec4272"],"kinds":[1]}

Full event that should be returned in both queries but returned only when querying by ID:

{
  "id": "8ba5ff1e7fa6313bf21edbff9fc930b1b85fb5cbb9333d6ac1ea61d4116c29d2",
  "pubkey": "68d81165918100b7da43fc28f7d1fc12554466e1115886b9e7bb326f65ec4272",
  "created_at": 1716830422,
  "kind": 1,
  "tags": [
    [
      "e",
      "06269da7dbf8d9b6b1b14a9017c6a05e05c999b98e74c8e0aea1b6ba7d42911a",
      "",
      "root"
    ],
    [
      "e",
      "5c411e3f150ee3438dfc167d0e45bf062aa00164d123357ca4386f4eab81ed7b",
      "",
      "reply"
    ],
    [
      "p",
      "a008def15796fba9a0d6fab04e8fd57089285d9fd505da5a83fe8aad57a3564d"
    ],
    [
      "p",
      "68d81165918100b7da43fc28f7d1fc12554466e1115886b9e7bb326f65ec4272"
    ],
    [
      "p",
      "38e02d76b4299acea0c847217b602bce12a9d3cea3cabdfd616c58324cff73d3"
    ]
  ],
  "content": "Hey, I not tested on Alpine yet. Have you any issue?",
  "sig": "fac2770f5c294b4b99183746f905ca63ba1be6b6a6f87435a9a76791d0afbb7855f3d2be459ee9437b41093c8dff493fa9c0b63d1016ff9f6ec5f1f53418996d"
}

jb55 commented 4 days ago

hmm ok thanks for confirming, I will continue to look into this then.

it's strange to me because I do queries like this all the time with notedeck and I have no issues, but maybe there is an off-by-one index issue or something I haven't noticed...

rust-nostr / nostr

Some problems with database #454

Example