Open dougollerenshaw opened 3 years ago
I think the cache is keeping track of things in a local file, and it wasn't written in such a way that it can handle multiprocess collisions on r/w of that file. @danielsf does that sound right?
Yes. That is a correct assessment of cloud cache's capabilities, and is very likely what is happening.
Got it. So I suppose the answer is just "don't do that", right?
Alternately, is there already a local cache of all NWB files somewhere in the /allen
filesystem that would let me accomplish what I was trying to do without having to write a loop to download all NWB files serially?
yes, "don't do that" :) no, we have not made a local copy of the downloaded cache
I'm going to post this here, in case we decide we want to add code to support multithreaded downloading in the future. Apparently threadsafety of boto3 is a nontrivial question
https://emasquil.github.io/posts/multithreading-boto3/
(we might be okay because S3CloudCache
carries around a boto3 client, which seems to be the threadsafe part of the process)
Thanks @danielsf and @djkapner. I'll go ahead and write a loop to download all serially.
Feel free to close this issue if this isn't something you want to deal with, or leave it here as a future to-do if you'd like.
Describe the bug I'm getting
JSONDecodeError: Extra data: line 12 column 2 (char 2984)
when attempting to load NWB files from the VisualBehavior cache.This occurred after starting a process to download all NWB files to a local directory. I set out to do this to make it possible to do a summary analysis of all experiments. I was hoping to use the NWB files for this analysis in order to ensure that results were consistent with what an external users would get when doing the same analysis.
To Reproduce I did the following to start the download process of all BehaviorOphysExperiment NWB files using 16 cores on my local machine:
Expected behavior I expected this process to take some number of hours to complete. At the end, I expected all NWB files to be in the
data_storage_directory
defined aboveActual Behavior The process started running as expected. After approximately 20 NWB files had been downloaded, I got the following error:
Now, simply calling:
results in the same error as above for any
oeid
Environment (please complete the following information):
Additional context I'm assuming that the parallel processing has somehow corrupted the manifest file. Is this true? If so, is there some other way to download the NWB files beyond what I tried above? Should I simply download them in a serial loop and wait however long it takes?
This is also related to a recent forum question (https://community.brain-map.org/t/visual-behavior-optical-physiology/1183), so I suspect external users will run into similar problems if attempting to parallelize the download process.
Do you want to work on this issue? Yes, I'd like to help solve this.