Document expected errors for SQLITE3

[RichardoC]

Mind documenting which errors exactly SQLITE3 is expecting from certain edge conditions?

For example, when SQLITE3 tries to read bytes from an empty file I return io.EOF from my ReadAt implementation, but this fails with panic: unable to open database file: invalid argument

Thanks!

[psanford]

Hmm, I don't think the issue you are seeing is simply from returning io.EOF from ReadAt. For example the tests in this package start with empty vfs files and do ReadAt's on them. If we add a log statement to the ReadAt function in the test we can see that this works:

go test -v -count=1 -mod=mod
=== RUN   TestSqlite3vfs
readat offset=0 size=100 :: result n=0 err=EOF
readat offset=24 size=16 :: result n=0 err=EOF
readat offset=512 size=8 :: result n=0 err=EOF
readat offset=24 size=16 :: result n=16 err=%!s(<nil>)
readat offset=13312 size=8 :: result n=0 err=EOF
readat offset=24 size=16 :: result n=16 err=%!s(<nil>)
readat offset=13312 size=8 :: result n=0 err=EOF
readat offset=24 size=16 :: result n=16 err=%!s(<nil>)
readat offset=13312 size=8 :: result n=0 err=EOF
readat offset=24 size=16 :: result n=16 err=%!s(<nil>)
readat offset=0 size=100 :: result n=100 err=%!s(<nil>)
readat offset=0 size=4096 :: result n=4096 err=%!s(<nil>)
readat offset=24 size=16 :: result n=16 err=%!s(<nil>)
readat offset=8192 size=4096 :: result n=4096 err=%!s(<nil>)
readat offset=4096 size=4096 :: result n=4096 err=%!s(<nil>)
--- PASS: TestSqlite3vfs (0.03s)
PASS
ok      github.com/psanford/sqlite3vfs  0.036s

In terms of semantics, we expect you to follow the semantics documented in the io.ReaderAt interface.

Its certainly possible you are hitting a bug in sqlite3vfs. If you can convert the existing test TestSqlite3vfs to a minimal case demonstrating your issue, that might be the easiest way to figure out what is happening.

[psanford]

Looking at the code I did notice one potential issue in this read path based on sqlite's expected semantics: https://github.com/psanford/sqlite3vfs/commit/cec222788cc6d7049c97a7bd81cd80423229a59e

Please let me know if that fixes it for you. If you are still having the issue with this patch then a minimal test case that reproduces the issue would be helpful.

[RichardoC]

Thank for clarifying the expected behaviour!

I'll keep investigating and see if I can provide more details.

Currently all I know is that when I'm testing kube-sqlite3-vfs I get the above panic when calling db.Exec unless I prevent access to wal and journal files like you did in https://github.com/psanford/sqlite3vfshttp/tree/main/sqlite3http-ext. If I do that, and use the _journal=MEMORY parameter in the dataSourceName it works fine. My suspicion around this behaviour comes from the fact the panic is always after it tries to read 1 byte from the journal file that is empty.

Is there a way to get any logs from SQLITE3 itself, or anything else that might help me investigate?

[psanford]

The write ahead log (WAL) generally requires shared memory semantics which is not implemented by this library (the xShm functions in the sqlite vfs documentation). I expect most implementations to be going over the network where a shared memory api wouldn't work properly. It looks like thats what you are doing.

Does using the rollback journal instead of a WAL work?