search

data-preservation-programs / spade

A Fil-Network Storage-Proposal Management Service
Other
7 stars 1 forks source link

Retrieval Bot - Run Spade Integrations for Spade RB #38

Closed xmcai2016 closed 5 months ago

jcace commented 10 months ago

The code to do this is already merged in here https://github.com/data-preservation-programs/RetrievalBot/pull/28

However, the retrieval worker code is not ready yet. Pending this: https://github.com/data-preservation-programs/RetrievalBot/pull/33 to get merged first.

Once that one is merged, we can start running this integration. To do this, we need to add a step to the pipeline to run the Spadev0. I started a branch for this here: https://github.com/data-preservation-programs/RetrievalBot/pull/34

We need to set up a cron task to run it (once per day should be plenty) - not sure how to do this exactly but @xinaxu should know. The code needs to do this:

 ./spadev0_integration spadev0 --sources https://cargo.dag.haus/active_replicas.json.zst
xinaxu commented 10 months ago

Our AWS pipeline should take care of creating the docker image for this new integration To create a cron task, go to AWS EventBridge and create a schedule

jcace commented 9 months ago

Update: Need to integrate this with the new Spade retrieval bot

jcace commented 9 months ago

Still pending the new Bitswap worker to get merged into Spade RB repo.

jcace commented 9 months ago

For this task, make a new branch on the Spade Retrieval Bot repo. Copy what I've started on this branch: https://github.com/data-preservation-programs/RetrievalBot/pull/34 , it has the .env file templated out and some of the build commands too