Closed jbusecke closed 3 months ago
Up till 2012 went really fast
2018-2022 also went through in a reasonable time...maybe there is something wrong with one of the files from 2012-2018?
Or it is an issue when there are too many files in the recipe. Testing 2012-2018 now to figure that out.
I am leaning towards the fact that too many files somehow affect the outcome of this. I believe I have tested all files in different chunks at this point, and in smaller batches, they all went through.
2012-end worked well, going back on the start in increments now.
2006-end is taking forever again...
@jbusecke anything I can do to help here?
I am not quite sure what the pickoff point here was, but all work on this dataset has been moved to https://github.com/leap-stc/metaflux_feedstock
The daily metaflux recipe has been running for 3!!! days (cc @cisaacstern). Testing if this is related to e.g. only some of the files here.