dandi / dandisets

730 Dandisets, 807.1 TB total. DataLad super-dataset of all Dandisets from https://github.com/dandisets
10 stars 0 forks source link

Provide basic documentation about setup #374

Closed yarikoptic closed 9 months ago

yarikoptic commented 9 months ago

needed for @kabilar and Aaron to replicate

edit: current cron jobs I see (some are disabled temporarily... drives are migrating etc)

2 * * * * chronic flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron.lock bash -c '/mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron'
50 1 * * * chronic flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron.lock bash -c '/mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron --mode verify'
### 30 1 * * * chronic flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron.lock bash -c '/mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron-zarrs'
# TEMP  52 * * * * chronic flock -E 0 -e -n /home/dandi/.run/backup2datalad-populate-cron.lock bash -c '/mnt/backup/dandi/dandisets/tools/backups2datalad-populate-assets-cron'
#52 * * * * chronic flock -E 0 -e -n /home/dandi/.run/backup2datalad-populate-zarr-cron.lock bash -c '/mnt/backup/dandi/dandisets/tools/backups2datalad-populate-zarrs-cron'
jwodder commented 9 months ago

@yarikoptic What exactly do you want me to say about the cron jobs? How much should this part of the documentation be tied to our current setup?

yarikoptic commented 9 months ago

At least that that we have two types - one to populate repos, and another to do the transfer. Might be worth disclosing that we also separate ones with zarrs out since they take too long etc.

Current setup provides an example, so we could share details, but generally doesn't need to be "too tied"

jwodder commented 9 months ago

@yarikoptic

At least that that we have two types - one to populate repos, and another to do the transfer.

Wouldn't that be obvious from the fact that those are two different commands? I'm finding it difficult to say anything meaningful about how to set up cronjobs.

jwodder commented 9 months ago

@yarikoptic I've created #376 with what I've written on everything except cronjobs.