search

RMI-PACTA / docker

Docker images
https://2degreesinvesting.github.io/docker/
MIT License
1 stars 0 forks source link

Validate tags in all repos #14

Closed maurolepore closed 3 years ago

maurolepore commented 3 years ago

https://github.com/2DegreesInvesting/docker/issues/3#issuecomment-728225899

it should check what tags already exist in each of the repos (including itself/the docker repo), and then determine if the specified tag is available (i.e not already used in any of the repos) and valid/reasonable (i.e. greater than any of the existing tags in any of the repos)

--

I moved this enhancement to its own issue.

cjyetman commented 3 years ago

This would ideally be done before the repos are cloned, since cloning is an expensive operation.

cjyetman commented 3 years ago

some research... GitHub API can return a JSON list of all tags in a repo https://docs.github.com/en/rest/reference/repos#list-repository-tags

so this... https://api.github.com/repos/2DegreesInvesting/PACTA_analysis/tags returns something like...

[
  {
    "name": "0.0.10",
    "zipball_url": "https://api.github.com/repos/2DegreesInvesting/PACTA_analysis/zipball/0.0.10",
    "tarball_url": "https://api.github.com/repos/2DegreesInvesting/PACTA_analysis/tarball/0.0.10",
    "commit": {
      "sha": "e7f8ca4e182282425f2d4bc2a8617682c4ecc1c3",
      "url": "https://api.github.com/repos/2DegreesInvesting/PACTA_analysis/commits/e7f8ca4e182282425f2d4bc2a8617682c4ecc1c3"
    },
    "node_id": "MDM6UmVmMjQzMTYwNjY4OnJlZnMvdGFncy8wLjAuMTA="
  },
  {
    "name": "0.0.9",
    "zipball_url": "https://api.github.com/repos/2DegreesInvesting/PACTA_analysis/zipball/0.0.9",
    "tarball_url": "https://api.github.com/repos/2DegreesInvesting/PACTA_analysis/tarball/0.0.9",
    "commit": {
      "sha": "ba917312601b7360bb7674ba797f8f6c2ef90d21",
      "url": "https://api.github.com/repos/2DegreesInvesting/PACTA_analysis/commits/ba917312601b7360bb7674ba797f8f6c2ef90d21"
    },
    "node_id": "MDM6UmVmMjQzMTYwNjY4OnJlZnMvdGFncy8wLjAuOQ=="
  }
]

that would need to be parsed properly... and we'd have to deal with authorizing the curl command for the private repos

maurolepore commented 3 years ago

Awesome! I think this gets us pretty close:

➜  docker git:(master) pacta-map 'git -C $i ls-remote --tags --ref'
From git@github.com:2DegreesInvesting/pacta-data.git
52f69029afa312225e39c46e1a9bf84bc1dba14e    refs/tags/0.0.1
30128e6130be46ab14b54e1f6982dc81a7bd2af8    refs/tags/0.0.10
7e7826c8d92ab8a357b75400d984655ed3d44c41    refs/tags/0.0.2
64ba2ae4351b933967806661e4a520f9f65e6be9    refs/tags/0.0.3
98c5197221a89aeb8ed7663df281d43ffccef5b1    refs/tags/0.0.4
2ffba6311a6be338fb604759d3488b048f5b9c45    refs/tags/0.0.5
abb6ee62e302bba0703eaedd6d4205bbab2e228f    refs/tags/0.0.6
3d5cb5c5aceab743eb60d488e360ad14458a741f    refs/tags/0.0.7
ac0b7e86b7383a8f5d2acedca9f132f63fb28576    refs/tags/0.0.8
2be25a6741cbbb27017ae32fced382e8356fee71    refs/tags/0.0.9
From git@github.com:2DegreesInvesting/create_interactive_report.git
dcfb01f20bf5f05755e56ed2cd1f6b95379a92bc    refs/tags/0.0.1
d3fa29afb92d2e0ccbf81faa615d7e5e984188a8    refs/tags/0.0.10
89d3a48e379a307a8570a4e17a1b0764b6faebb7    refs/tags/0.0.2
a097b80698fcb15e270a7d2af5595d4e63b14b50    refs/tags/0.0.3
c7d262c20279e9c3cbaff3f3eb449f6375376b6a    refs/tags/0.0.4
93b36411751a713e9ed3a772597a9dc62b5386eb    refs/tags/0.0.5
3112087c9d351dc560144eaee0a0644515831602    refs/tags/0.0.6
efd5b4f5d0fc512b6aa1cbf957f715a43f0fd766    refs/tags/0.0.7
4793fb3c6bb97730250d19ca2ef1c7f7372d6040    refs/tags/0.0.8
b83749d3e910ef75f238e1a383e562630467b685    refs/tags/0.0.9
From git@github.com:maurolepore/StressTestingModelDev.git
d686a6d7d6a45aa77bac1d7118b989126a819364    refs/tags/0.0.1
0ed8a0f17d2ae5868f53965de553d5a66c9c430a    refs/tags/0.0.2
b66c7987d1fde66701a0a2fd076bb5dacc742b7a    refs/tags/0.0.3
5d15194e1ff914c1b93a3924a478a67b4fc99754    refs/tags/0.0.4
6ea42f1bb570b0f1fc8897bbe3203d55cc2973b5    refs/tags/0.0.5
6b0c7543d5ed3bdcd3b43de8a49d9560c8e50abe    refs/tags/0.0.6
48267a6baa2ba74953b0db086290836ee77ac346    refs/tags/0.0.7
58f4b62a3179fff2998218eae2e4f75906d885f3    refs/tags/0.0.8
From git@github.com:2DegreesInvesting/PACTA_analysis.git
d7c1a8fda4d653d1e7d843c99b0a82c27f61f682    refs/tags/0.0.0
b41c444f062155ef4fe55a0fd918c13e46faa279    refs/tags/0.0.1
5aafde04adfb424a64843bc9bb775053bbff50c4    refs/tags/0.0.10
9db16f89fb552cf452c7065d7d01d90e40c9a56e    refs/tags/0.0.2
68bc4775d1beea04f368a81dc589e70c1480d342    refs/tags/0.0.3
bf4c4f517941cc2d83bc19d8d0a3cf8830de4505    refs/tags/0.0.4
4ecbca5c9a5c36443e1c4317060cd1d253501db6    refs/tags/0.0.5
5e627fe7d76a6905838255ea092c5f9ac0a1592a    refs/tags/0.0.6
067864904471e09145dab35020818695c02f8c17    refs/tags/0.0.7
d79f7f586fc3e19cc02df1ee063a3670dc055a8d    refs/tags/0.0.8
facc7ce742467e35008b69d4c3d167adc32d2b9a    refs/tags/0.0.9
cjyetman commented 3 years ago

for context, I was looking into ways of determining if a given tag is available in all the repos before cloning them locally, since cloning is an expensive (time and space) process, and if the specified tag is not available, then essentially you have to start the whole process over

maurolepore commented 3 years ago

Thanks! Yes, I think I understand your intent. git ls-remote seems to not need a local clone (above, the unfamiliar command pacta-map is only iterating over a list of URLs -- my example did not show the URL explicitely in the command but it comes implicitely from the URL linked to each remote). See this other simpler example:

➜  ~ git ls-remote --tags --ref git@github.com:r-lib/waldo.git
a42dd6dc32ff710a65581552890b2e0b9337316f    refs/tags/v0.1.0
317b3888b32494f984ec542239b1c4e5cba483d3    refs/tags/v0.2.0
39de75e16e68ca63ec89e520a1ed6801dd98ef78    refs/tags/v0.2.1
e4d2d5d4e0f168e8ae88a17c0dedf43286a69770    refs/tags/v0.2.2
a4d0cdf127d6b89c78782970653ef2f9b59c2157    refs/tags/v0.2.3
cjyetman commented 3 years ago

wow.... cool!

cjyetman commented 3 years ago

verified...

CJs-MacBook-Pro-2:pacta_web.11 cj2dii$ git ls-remote --tags --ref git@github.com:r-lib/waldo.git
a42dd6dc32ff710a65581552890b2e0b9337316f        refs/tags/v0.1.0
317b3888b32494f984ec542239b1c4e5cba483d3        refs/tags/v0.2.0
39de75e16e68ca63ec89e520a1ed6801dd98ef78        refs/tags/v0.2.1
e4d2d5d4e0f168e8ae88a17c0dedf43286a69770        refs/tags/v0.2.2
a4d0cdf127d6b89c78782970653ef2f9b59c2157        refs/tags/v0.2.3
maurolepore commented 3 years ago

Even closer ...

➜  git pacta-map 'git ls-remote --tags --ref $i' | cut -d / -f 3 | sort | uniq
0.0.0
0.0.1
0.0.2
0.0.3
0.0.4
0.0.5
0.0.6
0.0.7
0.0.8
0.0.9
maurolepore commented 3 years ago

I'm drafting a solution now.