search

NDAR / nda-tools

Python package for interacting with NDA web services. Used to validate, submit, and download data to and from NDA.
MIT License
47 stars 22 forks source link

write to closed file #89

Open dmd opened 7 months ago

dmd commented 7 months ago

I very often am getting this:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/NDATools/Download.py", line 49, in run
    func(*args, **kargs)
  File "/opt/conda/lib/python3.11/site-packages/NDATools/Download.py", line 308, in write_to_download_progress_report_file
    download_progress_report.flush()
ValueError: write to closed file

If I run the program again, it works fine - usually.

gregmagdits commented 7 months ago

Are you running multiple instances of the downloader at the same time when you see this error?

dmd commented 7 months ago

Nope.

dmd commented 7 months ago

But I'm also failing to reproduce it any more, so maybe something was weird about my container. I'll reopen if it recurs.

liningpan commented 7 months ago

I think the reason is that this thread pool is not joined download_progress_file_writer_pool

https://github.com/NDAR/nda-tools/blob/e4069d4bc79c563f3825e7cec901f54cd3dd8556/NDATools/Download.py#L358-L361

You should only close the file handle after joining the download_progress_file_writer_pool. Otherwise the file handle could be invalidated.

This also means that the download report may be incomplete

gregmagdits commented 7 months ago

yea, I think that makes sense. Thank you for pointing that out. Eventually I think we are going to use a sqllite db instead of writing to csv files, but this should be a quick fix that we can include in the next release. Thanks again