Add directory crawler populator to yield STAC Collections and Items + CLI utility

[fmigneault]

Changes

• Add request session keyword to all request-related functions and populator methods to allow sharing a common set of settings (auth, SSL verify, cert) across requests toward the STAC Catalog.
• Add DirectoryLoader that allows populating a STAC Catalog with Collections and Items loaded from a crawled directory hierarchy that contains collection.json files and other .json/.geojson items.
• Add a generic CLI stac-populator that can be called to run populator implementations directly using command stac-populator run <implementation> [impl-args].
• Remove hardcoded verify=False to requests calls. If needed for testing purposes, users should use a custom requests.sessions.Session with verify=False passed to the populator, or alternatively, employ the CLI argument --no-verify that will accomplish the same behavior.

Testing

Run the following commands:

# get sample data
git clone https://github.com/ai-extensions/stac-data-loader /tmp/stac-eurosat

# get magpie cookie
curl \
    -k \
    -X POST \
    --cookie-jar /tmp/magpie-cookie.txt \
    -d '{"user_name":"...","password":"..."}' \
    -H 'Accept:application/json' \
    -H 'Content-Type:application/json' \
    'https://{hostname}/magpie/signin'

# run populator on data
stac-populator run DirectoryLoader \
    "https://{hostname}/stac/" \
    "/tmp/stac-eurosat/data/EuroSAT/stac/subset" \
    --update --prune \
    --no-verify --auth-handler cookie --auth-identity /tmp/magpie-cookie.txt

Results for reference:

• https://hirondelle.crim.ca/stac/collections/EuroSAT-subset-train
• https://hirondelle.crim.ca/stac/collections/EuroSAT-subset-validate
• https://hirondelle.crim.ca/stac/collections/EuroSAT-subset-test

[fmigneault]

@huard If you delete the directory (recursive) before make setup-pyessv-archive, does it work? The rm could be added before the git clone to make sure the error does not happen, but I prefer to leave it up to the user to avoid removing something they did not intend to wipe.

[fmigneault]

and if I go into the docker directory and do docker compose up, I'm getting errors like:

(stac) david@it-282:~/src/stac-populator/docker$ docker compose up
[+] Running 2/1
 ✔ Network docker_default            Created                                                                                                                                                          0.1s 
 ✔ Volume "docker_stac-db"           Created                                                                                                                                                          0.0s 
 ⠋ Container stac-populator-test-db  Creating                                                                                                                                                         0.0s 
Error response from daemon: Conflict. The container name "/stac-populator-test-db" is already in use by container "f701152d5d735273fc26e499b75a2bf00de839e08d18ef8f3a12c85efb5b7fda". You have to remove (or rename) that container to be able to reuse that name

I tried to remove that container, but then I get the same error with another hash.

Does adding --rm within the same command that does docker compose up work?

[huard]

I think I'd prefer something like this (not sure it's actually working):

PYESSV-ARCHIVE-DIR = ~/.esdoc/pyessv-archive

$(PYESSV-ARCHIVE-DIR)%:
        @echo [ -d $@ ] || git clone "https://github.com/ES-DOC/pyessv-archive" $@
        cd $@ && git pull

Not sure I understand where to put the --rm.

[fmigneault]

PYESSV-ARCHIVE-DIR = ~/.esdoc/pyessv-archive

$(PYESSV-ARCHIVE-DIR)%: @echo [ -d $@ ] || git clone "https://github.com/ES-DOC/pyessv-archive" $@ cd $@ && git pull

I think something similar could work.

Not sure I understand where to put the --rm.

My bad. I confused with docker run commands. Maybe try docker compose down --rmi then retry make docker-start.

[huard]

I think what was needed was docker container prune.