threefoldtech / grid_deployment

Deploy a full Grid backend with docker-compose and snapshots

Apache License 2.0

2 stars 0 forks source link

TF Hub Sync Script #77

Open Mik-TF opened 1 month ago

Mik-TF commented 1 month ago

Situation

The TF Hub procedure is mostly done but we need to:
- make sure the sync script keeps syncing after it's initial sync (not the case currently)
Tested so far: uploaded an flist to the master hub, and could not find it in the other two after a while

References

As explained by @coesensbert. @maxux can you have a look? Please write here if you have issues or things are blocking and we can solve this.

Mik-TF commented 1 month ago

Update

Maxime is currently investigating and testing the syncing process:
- zdb sync is working fine
- flist static files not being synced
WIP and should be fixed soon

@maxux feel free to add info here if you deem it relevant!

maxux commented 1 month ago

Script https://github.com/threefoldtech/grid_deployment/blob/development/grid-hub/hub-clone.py updated to latest version which does a way better job on syncing. I updated production hub as well to support improved up-to-date checking.

coesensbert commented 1 month ago

see: https://github.com/threefoldtech/tf_operations/issues/2831

coesensbert commented 1 month ago

new sync script and cmd applied on the two extra hub's

Mik-TF commented 1 month ago

@coesensbert does this mean this issue is fixed? Has the script been tested so we can close? Thanks.

coesensbert commented 4 weeks ago

Updated script was applied on the two extra hub's, but still need to test if sync is going well now

Mik-TF commented 3 weeks ago

Hi @coesensbert! Please let us know, in the following days if possible, how the sync went. Thanks

coesensbert commented 3 weeks ago

so now the updated scripts have been applied to the two extra hub's, I tested the following: uploaded an archive to hub.grid.tf, this flist appeared on both extra hub's: https://hub.grid.tf/linkmark.3bot/hub-sync-tst.flist.md https://hub.de.grid.tf/linkmark.3bot/hub-sync-tst.flist.md https://hub.us.grid.tf/linkmark.3bot/hub-sync-tst.flist.md

However, I could not confirm the metadata did the same sync between the servers since:

metadata on DE and US hub point to zdb://hub.grid.tf:9900/default
did not see a data increase on both the DE or US hub. I also can not test a deploy from these hubs since ZOS does not support it yet to use a different hub then hub.grid.tf.
```
root@Main-Grid-DE-hub-02 ~ # du -sh /srv/*
5.1G    /srv/0-bootstrap
760G    /srv/0-db_data
609M    /srv/0-db_index
3.8G    /srv/0-hub_public
20K     /srv/0-hub_workdir
```

Mik-TF commented 3 weeks ago

@maxux @xmonader @sabrinasadik

Any idea?

How to test from these hubs instead of the official hub.grid.tf
How to confirm the metadata

Thanks

Mik-TF commented 3 weeks ago

@maxux

Thabet proposes this: "the verification should be running another hub and execute that sync script"

If you can have a look next week, when you're back at work. Thanks!

Mik-TF commented 2 weeks ago

@maxux please give status of the issue and it anything is blocking. Thanks!

Mik-TF commented 2 weeks ago

@xmonader can you please give more info on what you mean here? "the verification should be running another hub and execute that sync script"

So Maxime can move forward with this. Thanks!

EDIT: Had a talk with Maxime. He might be OK. He should reply to this issue soon.

maxux commented 2 weeks ago

About metadata: on the main hub, metadata are written in the .flist including the backend (which is supposed to be used by zos at some point). Right now, flist are copied as-it from the main hub and metadata are not updated. I should add an extra step while syncing which update metadata as well to apply local hub backend. I'll see how properly implement that.