Make sure that the file stream for tempfiles are closed successfully and files are written correctly before passing the files to a planner. Use the file only after closing it like this. Additional checks for the files might be helpful also. This issue is well known (https://github.com/python/cpython/issues/115113).
It is an expected behavior. Path.write_text() is not an atomic operation. First it open a file for writing (creating a new file if it did not exist and truncating and existing file), then write an encoded string (perhaps in several chunks if it is large enough, the closes the file). If you read a file between its opening and writing into it, you can see an empty file or a partial content.
If you want an atomic operation, write into another file (preferable in the same directory), and then replace the target file with it (using os.replace()). The success is platform and filesystem depending, but it should work on Posix platform in common cases.
jkeskingvillage
commented
2 months ago
Make sure that the file stream for
tempfiles are closed successfully and files are written correctly before passing the files to a planner. Use the file only after closing it like this. Additional checks for the files might be helpful also. This issue is well known (https://github.com/python/cpython/issues/115113).See: https://github.com/IBM/nl2flow/blob/main/nl2flow/plan/planners.py#L122