search

turbomam / biosample-xmldb-sqldb

Tools for loading NCBI Biosample into an XML database and then transforming that into a SQL database
MIT License
0 stars 1 forks source link

improve chaining of `pre-basex-all`, `basex-all` and `postgres-all` targets #1

Closed turbomam closed 6 months ago

turbomam commented 6 months ago

basex-all includes load-biosample-sets, but make determines $(BIOSAMPLE-SET-XML-CHUNK-NAMES) when the the make process is first started, at which point the directory is empty, because the Biosample XML file hasn't been downloaded, unpacked,or split yet.

Switch to a target/recipe that get the directory listing

BIOSAMPLE-SET-XML-CHUNK-FILES=$(shell ls shared-chunks)

BIOSAMPLE-SET-XML-CHUNK-NAMES=$(notdir $(basename $(BIOSAMPLE-SET-XML-CHUNK-FILES)))

biosample_set_from_%:
    echo $@
    date
    time docker exec -it biosample-basex basex -c "CREATE DB $@ basex/shared-chunks/$@.xml"

load-biosample-sets: $(BIOSAMPLE-SET-XML-CHUNK-NAMES) # 5 hours? could possibly do in parallel on a big machine
turbomam commented 6 months ago

Maybe this as an interim solution that runs from A to Z with no hand-holding?

make pre-basex-all && make basex-all postgres-all