HMDB Large File Size Issue Preventing Git Push

[jmillanacosta]

Running this workflow: https://github.com/sec2pri/mapping_preprocessing/actions/runs/5821698417/workflow The generated datasources/hmdb/hmdb_metabolites_split.zip is 2159.84 MB in size. This exceeds GitHub's file size limit of 100.00 MB, resulting in the push being rejected.

https://github.com/sec2pri/mapping_preprocessing/actions/runs/5821698417/job/15784672195#step:4:38

remote: error: Trace: 3c8b7413c5532418c302f057ba37ad74c211f687cd2095b6f955d05ef993f701        
remote: error: See https://gh.io/lfs for more information.        
remote: error: File datasources/hmdb/hmdb_metabolites_split.zip is 2159.84 MB; this exceeds GitHub's file size limit of 100.00 MB        
remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com./        
To https://github.com/sec2pri/mapping_preprocessing
 ! [remote rejected] main -> main (pre-receive hook declined)
error: failed to push some refs to 'https://github.com/sec2pri/mapping_preprocessing'
Error: Process completed with exit code 1.

Do we need to store this data? Should we preprocess before pushing? Which data is needed from this XML?

[tabbassidaloii]

Yes, we can proceed with the Java script. we need hmdb_metabolites_split.zip, but the scripts need a few changes before running. I will notify you when I push it

[tabbassidaloii]

@jmillanacosta because of the size of the files, we won't push the output files in GitHub

[DeniseSl22]

Some more steps which need to be included in the HMDB job:

• [ ] Convert current github action bash commands to separate bash script (for running locally and updating Shiny app)
• [ ] Include checking if data (with same name) is already available (for running locally and updating Shiny app)
• [ ] Add running .jar processing file for HMDB data to Github Action
• [ ] Create a logfile to save (unique) file content
• [ ] Push logfile to github
• [ ] If new release/running job again, compare to existing log file.

[tabbassidaloii]

@jmillanacosta