earthcube / geocodes

This is the containers stacks to run geocodes
0 stars 0 forks source link

Summary workflow (blocker) #12

Closed valentinedwv closed 1 year ago

valentinedwv commented 1 year ago

specifics:

MBcode commented 1 year ago

cleaning up and documenting it more now, will get in shared place soon, linked from this issue

MBcode commented 1 year ago

starting diagram / documentation of summary workflow https://github.com/MBcode/ec/blob/master/summary.md

MBcode commented 1 year ago

The 1st part of getting a crawl uses a workaround to get quads/repo for the summarization

valentinedwv commented 1 year ago

So, we need to add a note not to disable to milled:true to the doco

I was thinking about disabling the default templates for glcon configuration no longer run milled gleaner, because Nabu loads the data to the graph.

MBcode commented 1 year ago

btw the fix_runX.sh process can take summoned as well, it just takes longer; then it makes the quads, and now materialize_summary.py takes the repo.nq to repo.tll for uploading. Just for safety sake, I think it is best to upload both of these files, to make sure the graph-urns are exactly the same btwn them. The latest PR gets this all going w/blaze vs fuseki, and later we still have the option of query of file via lib vs using any endpoint at all

valentinedwv commented 1 year ago

complete on dev https://github.com/earthcube/earthcube_utilities/tree/dev/summarize