search

AlecThomson / arrakis

BSD 3-Clause "New" or "Revised" License
2 stars 1 forks source link

RMExtract related issues and notes #28

Open tjgalvin opened 1 year ago

tjgalvin commented 1 year ago

When running the FRion correction stage of the arrakis pipeline a number of issues internal to RMextract were found. This issue aims to record them for future generations.

The compute nodes of petrichor do not have external network access. Internally, RMextract attempts to download data files from remote sites over ftp. This fails. @AlecThomson found that by supplying dummpy proxy settings through to RMextract it could be convinced to use a caches copy of these files, so long as the server url is of the form file:///path/to/ionex/files.

In this approach a secondary issue was found where the data file is copied into place and extracted using a subprocess call to gunzip if the file ends in .Z. Although this works when in a single-process environment, the concurrently running tasks would each try to download (i.e. copy from local cache) and extract the same file. This results in race-conditions where the file is constantly overwritten, and either gunzip fails outright or the extracted file is not complete. I added a collection of os.path.exists type checks and early escapes, which seems to get me by.

We should merge these fixes together and have a static fork?

AlecThomson commented 1 year ago

Thanks for documenting this @tjgalvin. I've got an open PR (perhaps prematurely) on RM-extract. Lets use my fork and combine our changes there: https://github.com/AlecThomson/RMextract

Then I'll open a issue on the main fork and see if the maintainers are interested

tjgalvin commented 1 year ago

I've opened up a race branch that tries to capture the file exists checks with some slight cleaning up. I think this should be merged after a review.

We will still need to add some exposed option to provide a formatter lookup.