Enable offline runs of EWCE with ExperimentHub

[Al-Murphy]

Enable EWCE to run offline by using local cache of ExperiementHub. Issue was highlighted here by a user.

This will require:

• Updating ewceData so that the neurogenomics/ewceData/R/utils.R and the get_ExperimentHub function along with all dataset functions handle a localHub=T parameter
• Update all EWCE functions that call and load datasets from ewceData with the parmeter. The parameter will need to propagate up through the functions.
• Add a test case
• Push to Bioconductor

[chrisclarkson]

Hi Thank you for making this resource available! I too work on a cluster that is closed off from the internet (which is obviously beyond your control)... which is making it difficult to use your package: I tried testing the ExperimentHub problem directly (with localHub=T):

and localHub=F:

and so I tried downloading the sqlite database directly:

And it seems to have worked...

Looking inside the ~/.cache/R/ExperimentHub/ databases: It seems that certain data might be missing from the resource table. But the downloaded experimenthub.sqlite data seem to be intact(?):

Hence I am wondering if there is some way that I can just add the experimenthub data manually to your BiocFilecache database to get the package working for me....?

A lofty ask- I know- but appreciate any advice you can give. Kind Regards

[Al-Murphy]

Hey,

Thanks for your message. I have been meaning to make the change so you can run EWCE offline with the localHub approach for some time now so this has convinced me to do it. I will update this issue once it's done.

In terms of your ask though, I don't think we could/should add this data to the BiocFileCache given that adding these to the experimenthub database is the best practice and gives use a single location for version control, access etc. When I do make the change to enable localHub offline runs I think your best option is to get on to the bioconductor maintainer of experimenthub to find out how best to cache a local version of the files you need and copy them to a location in your closed cluster where they can be accessed when you run your analysis. Otherwise, it might be worth running this step locally, often EWCE runs don't use a lot of RAM although this does vary vastly on the dataset.

Hope this helps! Alan.

[chrisclarkson]

Thanks for the quick reply- so ultimately I should be able to copy some data onto my local cluster and use one of your inhouse EWCE commands to add these data in a way that I can then analyse them with EWCE::plot_mean etc. ?

[Al-Murphy]

The necessary changes have now been made (>= v 1.7.1) to run EWCE offline - the instigated functions have been documented, I've also updated the vignettes so check these for more info.

On you issue, I'm not sure this will work as I haven't done it before (why I think you should get in touch with the maintainers of Experimenthub) but it makes sense to me since the caching stores a version of the experimenthub database that you could just copy this from your local machine to your HPC. You would need to be sure your HPC experimenthub call is pointing to the correct location.

I'm going to close this issue since this is all we can do from our end with EWCE but do feel free to reopen if you have any issues running EWCE with localhub = TRUE