Closed PietrH closed 2 months ago
These errors have become much more rare now I'm taking more care not to write to the same file with multiple workers, but I'm still getting synchronicity errors.
2024-06-11 11:59:33.668407 [WARN] skipping due to Python error: Unable to synchronously open file (truncated file: eof = 12288, sblock->base_addr = 0, stored_eof = 25694)
Warning message:
In py_run_string_impl(code, local, convert) :
OSError: Unable to synchronously open file (truncated file: eof = 12288, sblock->base_addr = 0, stored_eof = 25694)
Run `reticulate::py_last_error()` for details.
2024-06-11 12:04:25.794271 [WARN] skipping due to Python error: Unable to synchronously open file (truncated file: eof = 4096, sblock->base_addr = 0, stored_eof = 25432)
Maybe these are unrelated?
On my end, there should never be multiple workers trying to open the same file at the same time, with the current workflow, every hdf5 file is only opened once in the whole run.
On second viewing, this error seems to originate from the OS/filesystem, so probably unrelated to vptstools
In that case, this was fixed by me removing overlapping date intervals/file sequences in my worker input data file. And this issue can be closed.
On a recent run on the HPC I noticed a bunch of Python errors in my error logs. After some searching I believe these might be due to a bug in the multiprocessing implementation of
vptstools.vpts
. This code was running on Tier-1 supercomputer infrastructure, so it's all a bit different to what I'm used to on a local system, sorry if this makes this more difficult to diagnose.This is what I'm seeing (just a sample, the log file is too big for Github):
There is thousands of these in a large run.
I'm passing python code via reticulate, this is what I'm doing:
The code is running within the worker framework: https://worker.readthedocs.io/en/latest/ so the logs above are actually from different parallel executions based on a csv file with a list of parameters that are being fed via the CLI. As you can tell, there are also some instances where Python is trying to access the same file in parallel, that was my bad, and has been fixed. It's especially the
wrong B-tree signature
,No such file or directory
andinflate()
that worry me.Originally I believed this might be due to corrupted hdf5 files, but now I believe it might be due to a bug in vptstools due to this comment: https://github.com/vaexio/vaex/issues/625#issuecomment-1007414088, the whole thread is worth reading: https://github.com/vaexio/vaex/issues/625
If you need more information on what exactly I was running, let me know and I'll send you a copy of all the tooling, input files, full logs, etc.
In any case, I did check one (1) of the output files where I was seeing these errors for the input files, and it seems fine, and is checksum identical to the file I can generate locally (without Python errors). So I'm not too alarmed. But I'd still love a second opinion, and it would be a lot of work to actually check all of the files I have these python warnings for in the log.
I'm trying to keep an eye on it for future runs as well!