search

datalad / git-annex

A non-official clone of git-annex established for DataLad purposes. No PRs will be merged, but could be used to test perspective git-annex patches. Official git-annex repository: https://git.kitenet.net/index.cgi/git-annex.git/
16 stars 3 forks source link

Set up for testing git-annex on non-GitHub clients #102

Closed jwodder closed 2 years ago

jwodder commented 2 years ago

Closes #57.

To do:

jwodder commented 2 years ago

@yarikoptic Please create a datalad/git-annex-ci-client-jobs repository and give me write permission so I can set up the badges-and-result-* repository.

Also, a GitHub token for writing to the new repository needs to be added to this repository as a secret named CLIENT_JOBS_GITHUB_TOKEN.

yarikoptic commented 2 years ago

@yarikoptic Please create a datalad/git-annex-ci-client-jobs repository and give me write permission so I can set up the badges-and-result-* repository.

I setup that repo, but do we need one more repo? I thought we could just use branches in that singular one for badges too

Also, a GitHub token for writing to the new repository needs to be added to this repository as a secret named CLIENT_JOBS_GITHUB_TOKEN.

you are admin in both. Please create one and share (privately)?

jwodder commented 2 years ago

@yarikoptic

I setup that repo, but do we need one more repo? I thought we could just use branches in that singular one for badges too

I'm setting this up so that, when the script runs on a client, it pulls the latest configuration from this repository, which means it has to run in a clone of this repository. Thus, if the script also pulled build-* branches from the same repository, it'd be constantly switching between branches, and things would be needlessly complicated. The alternative would be to use a separate clone of datalad/git-annex for the build-* branches, but at that point it's just cleaner and barely makes a difference to use a separate repo.

yarikoptic commented 2 years ago

note: the same repo could be cloned/checked out into the same/different branches in an infinite number of locations ;-)

yarikoptic commented 2 years ago

note: I was arguing only against (somewhat) creating badges-and-result-* repository of some kind you mentioned in addition to git-annex-ci-client-jobs (already created). May be you just meant "branches", not "repository" in

@yarikoptic Please create a datalad/git-annex-ci-client-jobs repository and give me write permission so I can set up the badges-and-result-* repository.

?

jwodder commented 2 years ago

@yarikoptic I did mean "repository"; git-annex-ci-client-jobs is the build-*, result-*, and badges repository, and nothing else, and there isn't a need for yet another repository.

On further consideration (including the fact that GitHub tokens can't be scoped to a single repository), I may just use datalad/git-annex for all of the branches & badges and not bother with git-annex-ci-client-jobs.

yarikoptic commented 2 years ago

On further consideration (including the fact that GitHub tokens can't be scoped to a single repository), I may just use datalad/git-annex for all of the branches & badges and not bother with git-annex-ci-client-jobs.

my only concern so far against that would be that those build- branches would contain .deb thus making this repo then quite heavy and possibly slowing its own CI clones etc. But I guess it could be non-substantiated.

jwodder commented 2 years ago

@yarikoptic So should we use a separate repository or not?

yarikoptic commented 2 years ago

@yarikoptic So should we use a separate repository or not?

if it is easier to go without a separate one with no impact on other operation of datalad/git-annex workflows (e.g. would main cron job get "cloning" of the repo very much slower to start with?) -- using one for everything is ok with me. Your choice.

  • a) using a personal token for an account (I'm not using my account for that; is there a datalad-bot?)

there is https://github.com/datalad-tester we could "recover" if needed ;-)

or (b) adding a repo-specific SSH public key to datalad/git-annex-ci-client-jobs and using the private key as a workflow secret.

would be as fine if needed.

yarikoptic commented 2 years ago

@yarikoptic yarikoptic marked this pull request as ready for review 2 days ago

ha -- I don't remember doing that in my clear consciousnesses ;) It is still WiP, right @jwodder ?

jwodder commented 2 years ago

@yarikoptic Yes. I've decided to use separate repositories, and I've set up a deploy key for cloning. There's a test build running right now. After that, I still need to do the items listed in the initial comment.

jwodder commented 2 years ago

@yarikoptic Do you have a preferred GPG key to use for signing the build-* commits, or should I just create a fresh one? Also, exactly how should the clients react if a commit lacks a valid signature?

jwodder commented 2 years ago

@yarikoptic Also, I want to set up tinuous to collect logs & artifacts from datalad/git-annex-ci-client-jobs, and I want to set it up next to the tinuous setup for the datalad/git-annex logs, but I don't remember where that is. It doesn't seem to be under dandi@drogon's account.

yarikoptic commented 2 years ago

@yarikoptic Do you have a preferred GPG key to use for signing the build-* commits, or should I just create a fresh one?

We already have one in secrets of this repo - DATALAD_BUILDER_GPGKEY .

Also, exactly how should the clients react if a commit lacks a valid signature?

I guess Error out would be the safest? I do understand that it might lead to the flood of CRON error notifications since then we would not "react" to present build- commits, but I don't see any other "safe" way, right?

next to the tinuous setup for the datalad/git-annex logs, but I don't remember where that is.

that would be great! drogon contains only dandi related tinuous etc setups. DataLad related setups are all under datalad@smaug:/mnt/datasets/datalad/ci/

jwodder commented 2 years ago

@yarikoptic

I guess Error out would be the safest?

I meant, should it exit immediately or continue with other build-* branches? And should it delete the invalid local and/or remote branch?

yarikoptic commented 2 years ago

I guess we would iron out gory details after trying and seeing. I would rely on your analysis of desired logic, meanwhile

I meant, should it exit immediately or continue with other build-* branches?

I guess "other" in the order sorted by version?

And should it delete the invalid local and/or remote branch?

I guess without deletion we would go in an infinite loop... I feel that we eventually should introduce some way to "protocol" what was tried so we don't come back to it... may be we should push build-*-skipped ref of some kind with logs etc so we could keep skipping them and leave to admin to analyze and either to remove all those build-* branches or just skipped after fixing up what needs to be fixed ?

jwodder commented 2 years ago

@yarikoptic I decided to make the script handle invalid signatures by renaming the remote branch to invalid-{clientid}-{buildid} and deleting it locally. That way, invalid branches are kept around for inspection but don't keep triggering new jobs.

jwodder commented 2 years ago

@yarikoptic I believe the only part left now is to actually test running a client on ndoli. Should I set the client up under my account or some other account?

yarikoptic commented 2 years ago

@yarikoptic I believe the only part left now is to actually test running a client on ndoli. Should I set the client up under my account or some other account?

Please setup under your account for now, meanwhile I will email admins on possible a dedicated one. with nfs4_setfacl you should be able also to allow me access to see (if so desired). my login d31548v .

jwodder commented 2 years ago

@yarikoptic FYI, the DATALAD_BUILDER_GPGKEY key expires in a month.

jwodder commented 2 years ago

@yarikoptic The client setup is working now, though I'm not going to actually create a cronjob until this branch is merged, as the code auto-updates from master. The only problem is that ndoli is very, very slow, and the tests keep hitting the one-hour timeout I set.

yarikoptic commented 2 years ago

The only problem is that ndoli is very, very slow, and the tests keep hitting the one-hour timeout I set.

eh, right -- on the discovery main node under a POSIXy /dartfs/rc/lab/D/DBIC/DBIC/archive/tmp it takes 5576.99s so aiming for two hours :-/ kinda wasteful, but can we boost timeout to two hours?

but in general, if PR is ready -- time to take out of draft mode?

jwodder commented 2 years ago

@yarikoptic Ready for review.

yarikoptic commented 2 years ago

FWIW: seeking possible ideas from @joeyh on speeding tests battery up https://git-annex.branchable.com/todo/speed_up___34__standalone_build__34___and__47__or_tests/?updated

yarikoptic commented 2 years ago

meanwhile, could you please setup client testing on smaug. It should not be as "exciting" as discovery, but at least would give us a sample client to test / report on

jwodder commented 2 years ago

@yarikoptic Should the smaug tests run the tests in any specific directory?

jwodder commented 2 years ago

@yarikoptic Also, what user account should I set this up under?

yarikoptic commented 2 years ago

probably as datalad user and somewhere e.g. like /mnt/datasets/datalad/git-annex-build-client?

jwodder commented 2 years ago

@yarikoptic Could you please merge this PR? Working with a different branch in a single-branch clone is very annoying.

yarikoptic commented 2 years ago

sure -- feel welcome to click Merge whenever you feel you want that state merged (I just see a fresh commit being tested).

jwodder commented 2 years ago

@yarikoptic Client set up on smaug.

yarikoptic commented 2 years ago

FWIW: seeking possible ideas from @joeyh on speeding tests battery up https://git-annex.branchable.com/todo/speed_up___34__standalone_build__34___and__47__or_tests/?updated

@joeyh fixed running standalone tests which might provide drastic boost! Let's see the timing on tomorrow build.