No profile shown after switching from Patchwork to Oasis backend

skyebend commented 3 years ago

What's the problem you want solved?

When Oasis runs its own server, it doesn't seem to know my identity, or show names on any posts. No error messages are logged.

~/oasis$ oasis
  oasis No configuration file found at /home/skyebend/.config/oasis/default.json, using built-in default values. +0ms
  oasis Listening on http://localhost:3000 +497ms
  oasis Starting Scuttlebutt service +0ms
ssb-friends: stream legacy api used
ssb-friends: get legacy api used
about-name-warmup: 3319.234ms

Context: I have been using Oasis in conjunction with Patchwork for some time. After turning off Patchwork and restarting Oasis and waiting for indexing to complete, no content appears in Profile or Mentions views (but I previously have lots of content when I run it with Patchwork providing ssb-server). I can see content in Latest and Extended view, but all posts are labeled with the short key instead of the name of the poster.

This is on master branch of Oasis, on Ubuntu with all indexes showing 100%. (Image Search does work tho and it is very fast!). I see the same behavior when running oasis or oasis-desktop. Perhaps there is something funny about my .ssb db?

black-puppydog commented 3 years ago

This is what I'm seeing too (see my comments in #534) and actually just signed in here to report I'm seeing this on both the pinephone and the raspberry pi at the moment. Both of them without patchwork running.

I propose to keep this issue here because it describes it much clearer than the other one and may be distinct.

christianbundy commented 3 years ago

I kinda wonder whether a db rebuild might fix this. A few months ago I added an ssb.rebuild() method to FlumeDB/SSB-DB, maybe we should use it here?

christianbundy commented 3 years ago

FWIW I tried downgrading to SSB-Backlinks 1.x.x and it renegerated the backlinks view and the problem went away. I upgraded to SSB-Backlinks 2.x.x and it regenerated again and the problem wasn't there anymore. No idea why this would fix anything, but it might be a hint (or a red herring lol).

black-puppydog commented 3 years ago

I've seen this on two devices now, and at least on the pinephone, removing all flume indexes and then re-indexing seemed to have fixed the problem, but letting the phone lie idle for a day (not touching it) made it reappear.

christianbundy commented 3 years ago

letting the phone lie idle for a day (not touching it) made it reappear

this is nightmare fuel

christianbundy commented 3 years ago

Are there any errors in the console that might signify this? I run Oasis via systemctl and found some logs with:

$ journalctl --user --unit oasis
-- Logs begin at Mon 2020-11-23 19:33:25 PST, end at Sun 2020-11-29 16:34:57 PST. --
Nov 24 07:38:15 orion zsh[70715]: updating 17129 feeds
Nov 24 07:38:32 orion zsh[70715]: transpose-name-index: 16673.396ms
-- Reboot --
Nov 24 07:43:48 orion systemd[652]: Started Oasis - Friendly Scuttlebutt Web Interface.
Nov 24 07:43:48 orion zsh[659]: 2020-11-24T15:43:48.897Z oasis No configuration file found at /home/christianbundy/.config/oasis/default.json, using built-in default values.
Nov 24 07:43:50 orion zsh[659]: 2020-11-24T15:43:50.727Z oasis Listening on http://localhost:4515
Nov 24 07:43:50 orion zsh[659]: 2020-11-24T15:43:50.734Z oasis Starting Scuttlebutt service
Nov 24 07:43:50 orion zsh[659]: ssb-friends: stream legacy api used
Nov 24 07:43:56 orion zsh[659]: could not retrive msg: Error [NotFoundError]: Key not found in database [@PDgpocIwRX5gpEHaIBTal2wl9bDb+Liy4+XhjDo4dq8=.ed25519,15]
Nov 24 07:43:56 orion zsh[659]:     at /home/christianbundy/.npm-global/lib/node_modules/@fraction/oasis/node_modules/levelup/lib/levelup.js:188:15
Nov 24 07:43:56 orion zsh[659]:     at /home/christianbundy/.npm-global/lib/node_modules/@fraction/oasis/node_modules/encoding-down/index.js:75:21
Nov 24 07:43:56 orion zsh[659]: could not retrive msg: Error [NotFoundError]: Key not found in database [@ILLFkMcjUkc6KgMKYO0c3ByVqBLjG2XwlE1k45tIc5Y=.ed25519,9]
Nov 24 07:43:56 orion zsh[659]:     at /home/christianbundy/.npm-global/lib/node_modules/@fraction/oasis/node_modules/levelup/lib/levelup.js:188:15
Nov 24 07:43:56 orion zsh[659]:     at /home/christianbundy/.npm-global/lib/node_modules/@fraction/oasis/node_modules/encoding-down/index.js:75:21
Nov 24 07:43:56 orion zsh[659]: could not retrive msg: Error [NotFoundError]: Key not found in database [@1sw77BWBwCzMlQuPG0YkZw8uiHRpEeBe5uLjzXGY36w=.ed25519,39]
Nov 24 07:43:56 orion zsh[659]:     at /home/christianbundy/.npm-global/lib/node_modules/@fraction/oasis/node_modules/levelup/lib/levelup.js:188:15
Nov 24 07:43:56 orion zsh[659]:     at /home/christianbundy/.npm-global/lib/node_modules/@fraction/oasis/node_modules/encoding-down/index.js:75:21
Nov 24 07:44:00 orion zsh[659]: about-name-warmup: 9688.906ms
Nov 24 07:44:00 orion zsh[659]: updating 17129 feeds
Nov 24 07:44:02 orion zsh[659]: could not retrive msg: Error [NotFoundError]: Key not found in database [@1sw77BWBwCzMlQuPG0YkZw8uiHRpEeBe5uLjzXGY36w=.ed25519,39]
Nov 24 07:44:02 orion zsh[659]:     at /home/christianbundy/.npm-global/lib/node_modules/@fraction/oasis/node_modules/levelup/lib/levelup.js:188:15
Nov 24 07:44:02 orion zsh[659]:     at /home/christianbundy/.npm-global/lib/node_modules/@fraction/oasis/node_modules/encoding-down/index.js:75:21
Nov 24 07:44:07 orion zsh[659]: could not retrive msg: Error [NotFoundError]: Key not found in database [@1sw77BWBwCzMlQuPG0YkZw8uiHRpEeBe5uLjzXGY36w=.ed25519,39]
Nov 24 07:44:07 orion zsh[659]:     at /home/christianbundy/.npm-global/lib/node_modules/@fraction/oasis/node_modules/levelup/lib/levelup.js:188:15
Nov 24 07:44:07 orion zsh[659]:     at /home/christianbundy/.npm-global/lib/node_modules/@fraction/oasis/node_modules/encoding-down/index.js:75:21
Nov 24 07:44:10 orion zsh[659]: transpose-name-index: 10107.156ms
Nov 24 07:44:13 orion zsh[659]: could not retrive msg: Error [NotFoundError]: Key not found in database [@1sw77BWBwCzMlQuPG0YkZw8uiHRpEeBe5uLjzXGY36w=.ed25519,39]
Nov 24 07:44:13 orion zsh[659]:     at /home/christianbundy/.npm-global/lib/node_modules/@fraction/oasis/node_modules/levelup/lib/levelup.js:188:15
Nov 24 07:44:13 orion zsh[659]:     at /home/christianbundy/.npm-global/lib/node_modules/@fraction/oasis/node_modules/encoding-down/index.js:75:21
Nov 24 07:44:19 orion zsh[659]: could not retrive msg: Error [NotFoundError]: Key not found in database [@1sw77BWBwCzMlQuPG0YkZw8uiHRpEeBe5uLjzXGY36w=.ed25519,39]
Nov 24 07:44:19 orion zsh[659]:     at /home/christianbundy/.npm-global/lib/node_modules/@fraction/oasis/node_modules/levelup/lib/levelup.js:188:15
Nov 24 07:44:19 orion zsh[659]:     at /home/christianbundy/.npm-global/lib/node_modules/@fraction/oasis/node_modules/encoding-down/index.js:75:21
Nov 24 07:44:25 orion zsh[659]: could not retrive msg: Error [NotFoundError]: Key not found in database [@1sw77BWBwCzMlQuPG0YkZw8uiHRpEeBe5uLjzXGY36w=.ed25519,39]
Nov 24 07:44:25 orion zsh[659]:     at /home/christianbundy/.npm-global/lib/node_modules/@fraction/oasis/node_modules/levelup/lib/levelup.js:188:15
Nov 24 07:44:25 orion zsh[659]:     at /home/christianbundy/.npm-global/lib/node_modules/@fraction/oasis/node_modules/encoding-down/index.js:75:21
Nov 24 07:44:31 orion zsh[659]: could not retrive msg: Error [NotFoundError]: Key not found in database [@1sw77BWBwCzMlQuPG0YkZw8uiHRpEeBe5uLjzXGY36w=.ed25519,39]
Nov 24 07:44:31 orion zsh[659]:     at /home/christianbundy/.npm-global/lib/node_modules/@fraction/oasis/node_modules/levelup/lib/levelup.js:188:15
Nov 24 07:44:31 orion zsh[659]:     at /home/christianbundy/.npm-global/lib/node_modules/@fraction/oasis/node_modules/encoding-down/index.js:75:21

I wonder whether y'all are getting those "could not retrive msg" errors too. :thinking:

skyebend commented 3 years ago

No console errors for me. I just switched back from Patchwork to Oasis with the intention of testing the rebuild index PR ... and somehow this time the re-index was very speedy, I didn't see the CPUs maxed out .. and all the names and Mentions showing normally :thinking: I had been trying to force upgrades of various npm dependencies. Maybe I had an old version of something that causes oasis's re-index to go badly? (Although I'm also not sure it is pulling new content, as Latest seems suspiciously old)

christianbundy commented 3 years ago

Highly recommend npm ci to reproduce the node modules. My first experience with this problem was when I was doing dependency upgrades, so my guess(/hope?) is that this is a problem that only crops up when you have weird node_modules.

Although I'm also not sure it is pulling new content, as Latest seems suspiciously old

If you go to settings you can do 'restart networking' to try to connect to some new peers, but this is a common-ish SSB problem. 😅

skyebend commented 3 years ago

sorry I didn't get a chance to test this before https://github.com/fraction/oasis/pull/540 merge. Today I switched from patchwork, saw the problem, pull master and was eventually able to see and use the index rebuild button. I'm not very familiar with node, It did take me a couple of attempts (npm install wasn't enough, needed npm ci to get a clean install) I can confirm that it did indeed rebuild the db, and I see my profile etc. Thank you!

skyebend commented 3 years ago

... also discovered that I'm seeing different versions of oasis when I start via oasis vs npm start in my oasis dir (the later runs the master branch) I'm assuming because I may also have global version, and the desktop as well.

christianbundy commented 3 years ago

I have a hunch that npm install is doing something funky sometimes, and that npm ci is necessary if you're upgrading Oasis from source. I've only ever seen this error when doing npm install and then doing Git gymnastics or editing package.json and re-running npm install. :thinking:

I'm assuming because I may also have global version, and the desktop as well.

Yep, npm start runs the version in your source code directory whereas oasis runs the globally installed version.

zelfos commented 3 years ago

Apologies in advance but I'm going to add more bad news to you Christian and all the Oasis collaborator: I'm experiencing the basically the same except I'm pretty sure the cause was the update and nothing to do with other clients like Patchwork. I had (only) used Oasis at least for the past 6 months without this problem until now. Let me explain my story:

Last week I updated to the latest unstable version and I notice some profiles weren't showing any posts but I also was seeing some of their posts on the summaries and other similar views. I was also having problems with hashtags just like in #534
I realize something is wrong so the next day I uninstall unstable Oasis and I proceed to install the latest stable version. This didn't fix my problem, though.
Next day I see this thread and see that someone fixed by removing the all flume indexes and reindexing so I proceed to do the same.
This was probably an error on my part cos' now I can't see anything and Oasis doesn't seem to be reindexing...EXCEPT it sometimes show a meesage tha some blobs could not be retrieved due to size limit exceed (which I have on my '.ssb' config) so I don't know what exactly is happening except I can't use Oasis.

I'm just posting this so you folx know. I have no problem with waiting for a fix specially with the holidays around the corner.

christianbundy commented 3 years ago

I can't see anything and Oasis doesn't seem to be reindexing...EXCEPT it sometimes show a meesage tha some blobs could not be retrieved due to size limit exceed

Thanks for the update. When you try to open localhost:3000 or whatever, what do you see?

zelfos commented 3 years ago

Basically the description of each view/page and nothing else where previously there was content. I forgot to mention that I had only left Oasis running by max. 1 hour. The .ssb/flume folder is only 184 K.

pic-selected-201206-1809-34

pic-selected-201206-1809-56

This is the usual console output:

~ oasis
  oasis No configuration file found at /home/zelfos/.config/oasis/default.json, using built-in default values. +0ms
  oasis Listening on http://localhost:3000 +21s
  oasis Starting Scuttlebutt service +0ms
ssb-friends: stream legacy api used
ssb-friends: get legacy api used
about-name-warmup: 817.904ms

zelfos commented 3 years ago

Ok, this is weird. I manually clicked start networking, restart networking and connect and sync, closed Oasis and launched it again and now Oasis had started the indexing process or whatever is called when the load bar of "X% of the messages are being processed" appears. But yesterday I did the same without results and Oasis was showing some peers to sync. Maybe I was connecting to unreliable peers/pubs?

Still, I don't quite understand why Oasis immediately started to retrieve message only after I closed it and restart it without any other action on my part. Well, at least now my Oasis should be ok, .sbb/flume is 600M and growing.

davegomez commented 3 years ago

I haven't been able to solve this issue and I can't make it to sync no matter what I do.

This is the latests error I'm getting in the terminal:

server error, from net:192.168.1.32:51661~shs:
Error: stream ended with:0 but wanted:64
    at drain (/Users/dave/.nvm/versions/node/v15.4.0/lib/node_modules/@fraction/oasis/node_modules/pull-reader/index.js:43:26)
    at /Users/dave/.nvm/versions/node/v15.4.0/lib/node_modules/@fraction/oasis/node_modules/pull-reader/index.js:63:18
    at /Users/dave/.nvm/versions/node/v15.4.0/lib/node_modules/@fraction/oasis/node_modules/pull-reader/index.js:20:7
    at drain (/Users/dave/.nvm/versions/node/v15.4.0/lib/node_modules/@fraction/oasis/node_modules/stream-to-pull-stream/index.js:126:18)
    at Socket.<anonymous> (/Users/dave/.nvm/versions/node/v15.4.0/lib/node_modules/@fraction/oasis/node_modules/stream-to-pull-stream/index.js:147:5)
    at Socket.emit (node:events:388:22)
    at TCP.<anonymous> (node:net:666:12)

christianbundy commented 3 years ago

I don't think that error is related to this problem.

@davegomez is it just Oasis running or is Patchwork running too? If you go to settings who does it say you're peering with? Need more info to help you!

davegomez commented 3 years ago

Hello @christianbundy, I'm running just Oasis, I don't use Patchwork.

In the settings, the only peer connection I can see is my Manyverse account and my Manyverse account can see my Desktop without issues.

Nevertheless, I can't get any data sync and my profile still appears empty (new)

If I click on Connect and Sync nothing happens and the console doesn't show anything else either.

Let me know what else I can do to try to debug the issue.

fraction / oasis

No profile shown after switching from Patchwork to Oasis backend #538

What's the problem you want solved?