keystore error: OS specific error: Cannot open file for writing

chair28980 commented 8 months ago

Running on Ubuntu 22.04.3.

When following the readme instructions, after successfully running ./register-rln.sh and then running docker-compose up -d the following error is triggered:

ERR 2023-12-18 18:15:24.561+00:00 failed to persist credentials topics="rln_keystore_generator" tid=1 file=rln_keystore_generator.nim:92 error="keystore error: OS specific error: Cannot open file for writing"

Ivansete-status commented 8 months ago

I think we need to change the current "write" approach: https://github.com/waku-org/nwaku/blob/shards-test-dogfooding/waku/waku_keystore/utils.nim#L30-L40

gabrielmer commented 8 months ago

Weekly Update

achieved: attempted many times to reproduce it in VMs. Did not succeed, everything worked.
next: define how to proceed. Either keep attempting to reproduce it, find someone that currently has the issue, or wait for someone to see it again.

vpavlin commented 8 months ago

@chair28980 Are you running it via Docker Desktop or "native" docker CLI? (IIRC @chaitanyaprem had an issue with volumes using Docker Desktop, not sure if it could be related)

chair28980 commented 8 months ago

@chair28980 Are you running it via Docker Desktop or "native" docker CLI? (IIRC @chaitanyaprem had an issue with volumes using Docker Desktop, not sure if it could be related)

native docker CLI

chair28980 commented 8 months ago

This might be due to the erroneous slash at the end of keystore.json here https://github.com/waku-org/nwaku-compose/blob/17444347ba742d2df52863d27c9954dc98bddd2b/docker-compose.yml#L60C24-L60C24

gabrielmer commented 8 months ago

I've been working the last week on reproducing the original issue without success. Attempted it in Ubuntu, macOs, Windows 10, Windows 11 vm, Windows 10 vm, from WSL, from CMD and probably some other variants. In all cases the keystore generation worked for me without any issues.

On the other hand, two users that had the write problem also had keystore.json created as a directory instead of as a file, which should be fixed by this. I'll ask one of them to try again with this new fix.

However, it's very likely that this isn't the root cause of the issue and just a separate bug.

gabrielmer commented 8 months ago

Weekly Update

achieved: more unsuccessful attempts to reproduce it in other environments, talking to users who have the issue to understand further
next: ask users with the issue to run with the new fix, see if it solves the issue either to some, all or none of them. Keep watching and tracking the issue closely

chaitanyaprem commented 7 months ago

I am running into same issue in one of my nodes on cloud. I am using ubuntu 23 and docker CLI to run.

ERR 2024-02-09 07:04:37.647+00:00 4/7 Mounting protocols failed topics="wakunode main" tid=1 file=wakunode2.nim:89 error="failed to mount waku RLN relay protocol: failed to mount WakuRlnRelay: exception in new WakuRlnRelay: could not parse the keystore: keystore error: OS specific error: Cannot open file for writing"

And i see that keystore.json is created as a directory instead of file.

ls -lrt keystore/
total 4
drwxr-xr-x 2 root root 4096 Feb  9 07:03 keystore.json

Do we have a work-around for this?

chaitanyaprem commented 7 months ago

I am running into same issue in one of my nodes on cloud. I am using ubuntu 23 and docker CLI to run.

ERR 2024-02-09 07:04:37.647+00:00 4/7 Mounting protocols failed topics="wakunode main" tid=1 file=wakunode2.nim:89 error="failed to mount waku RLN relay protocol: failed to mount WakuRlnRelay: exception in new WakuRlnRelay: could not parse the keystore: keystore error: OS specific error: Cannot open file for writing"

And i see that keystore.json is created as a directory instead of file.
ls -lrt keystore/
total 4
drwxr-xr-x 2 root root 4096 Feb  9 07:03 keystore.json
Do we have a work-around for this?

Looks like i have figured out the cause for this. In my case i did not do rln registration on this node and hence the keystore folder was not present. When i started nwaku using docker-compose it must have assumed since the path ./keystore/keystore.json is being mounted and is not available, it must be creating a directory instead of a file.

The error got fixed after i followed the following steps:

Delete keystore directory
Register RLN (which created the keystore dir and keystore.json file within it)
Start docker-compose

Since we have this statement in readme which makes it an optional step, this issue would be faced by others as well. Maybe we can create the keystore/keystore.json file in case it is not present, and check for empty credentials while loading this for initializing RLN.

This command will register your membership and store it in keystore/keystore.json. Note that if you just want to relay traffic (not publish), you don't need one.

chaitanyaprem commented 7 months ago

Maybe https://docs.docker.com/storage/bind-mounts/ can help take care of this situation.

cc @gabrielmer

gabrielmer commented 7 months ago

Makes sense! Thank you! I'll take a look into this :))

gabrielmer commented 7 months ago

So found out a couple things: 1- We should mount a volume for /keystore instead of /keystore/keystore.json as docker automatically creates the paths mounted as directories if they don't exist, as @chaitanyaprem mentioned

2- In nwaku when mounting RLN we try to read the file passed in the rln-relay-cred-path flag. This flag has a value by default in nwaku-compose but if the file doesn't exist, we try to open it anyway and get the OS specific error: Cannot open file for writing error. We should also check if the file actually exists before attempting to read from it

Opened PRs for both issues

gabrielmer commented 7 months ago

Seems to be fixed by https://github.com/waku-org/nwaku-compose/pull/47 :)

waku-org / nwaku-compose

keystore error: OS specific error: Cannot open file for writing #32