Add endpoint to get pipeline configuration file (and above all to get its parameters definition)

What I want, is that in the web app, the form to trigger the pipeline automatically adapts to the parameters, according the pipeline's configuration file :

https://circleci.com/docs/api/v2/#operation/getPipelineConfigById
[x] First, I will group all Routers that are about pipelines executions (listing them,triggering a pipeline execution) in one subfolder name pipeline-executions
[ ] I need one more endpoint to be able to get all git repos "all projects":
- they will be listed in the Web UI
- I also need to know for each git repo, if it was "set up to start building" : only those who are can have a pipeline execution triggered.
- Ok, here is the problem : On Circle CI API v2, there is no Endpoint to list all projects, at all. You can just get the infos of a given project, and that's it. You can list projects, (only those who you "follow"), usig Circle CI PAI v1, exactly the https://circleci.com/docs/api/v1/#projects Endpoint

SECRETHUB_ORG=gravitee-lab
SECRETHUB_REPO=cicd
# Nevertheless, I today think :
# Each team member should have his own personal secrethub repo in the [graviteeio] secrethub org.
# like this :
# a [graviteeio/${TEAM_MEMBER_NAME}] secrethub repo for each team member
# and the Circle CI Personal Access token stored with [graviteeio/${TEAM_MEMBER_NAME}/circleci/token]
# ---
export HUMAN_NAME=jblasselle
export CCI_TOKEN=$(secrethub read "${SECRETHUB_ORG}/${SECRETHUB_REPO}/humans/${HUMAN_NAME}/circleci/token")

export ORG_NAME="gravitee-lab3"

curl -X GET -H 'Content-Type: application/json' -H 'Accept: application/json' -H "Circle-Token: ${CCI_TOKEN}" https://circleci.com/api/v2/me | jq .

# https://circleci.com/docs/api/v1/#projects

curl -iv -X GET https://circleci.com/api/v1.1/projects  -H 'Content-Type: application/json' -H 'Accept: application/json' -H "Circle-Token: ${CCI_TOKEN}" | tail -n 1 | jq > all-followed-projects.json

# and to see this list is across all Github org,
# you can get the HTTP URL of each of those projects like this :  

curl -X GET https://circleci.com/api/v1.1/projects  -H 'Content-Type: application/json' -H 'Accept: application/json' -H "Circle-Token: ${CCI_TOKEN}" | jq ' .[].vcs_url'

# you can also get the list of all git branches for eahc project, like this : 
curl -X GET https://circleci.com/api/v1.1/projects  -H 'Content-Type: application/json' -H 'Accept: application/json' -H "Circle-Token: ${CCI_TOKEN}" | jq ' .[].branches' | jq 'with_entries(.value = (.value | type))'

Ok, so to sum it up, with this Circle CI API v1 Endpoint :
- I can get the list of all the repos I follow, (and since I follow them, then I know they are set up to start building... Ok I'll have to check it all anyway, but I can start listing all projects, and all github orgs I work with, starting from there
- I can get the github Orgs work with
- I can get the list of all git branches for each repo I follow

Now,this is where I Willneed Github OAuth :

The Circle CI API is pretty incomlplete, So to list all orgs, projects etc... I will use the Github API as source of truth
The user, just like on Circle CI, will have to grant access to the github Orgs he wants to work with
then it will use the Github API Token obtained from OAuth, to get list of all github repos, not just those which are followed, but only for granted access Github Orgs.
Then I will use Circle CI API v1to get all followed projects, the infered list of github orgs, and :
- label with the "cci_access_granted" label, each github org which was granted access retrieved with Github API,
- label with the "cci_followed" label, each git repos retrieved with Github API
Then I compare list of git branches, between Circle CI API v1 followed projects, and Github API :
- Github API :
  - to list organizations (with the Github OAuth token, only the granted access orgs will be returned) : https://api.github.com/user/orgs just like I did in ./src/server/lib/auth/permissions/pokus/github/index.ts line 112
  - to list repositories of an org : https://docs.github.com/en/rest/reference/repos#list-organization-repositories (or see the conexted repos
  - to list branches of a repository : https://docs.github.com/en/rest/reference/repos#list-branches
  - also, Whe I get the authentifcated user, I get a JSON response with many user releative URLs, especially for listing repos and github orgs (So I 'll use them,and use somthing else to get branches of each repos) :

2021-01-17T02:47:00.714259+00:00 app[web.1]: >>>>>>>>>>>>>>> GITHUB API RESPONSE (FOR AUTHENTICATED USER) IS :
2021-01-17T02:47:00.714448+00:00 app[web.1]: {
2021-01-17T02:47:00.714448+00:00 app[web.1]: login: 'Jean-Baptiste-Lasselle',
2021-01-17T02:47:00.714448+00:00 app[web.1]: id: 35227860,
2021-01-17T02:47:00.714449+00:00 app[web.1]: node_id: 'MDQ6VXNlcjM1MjI3ODYw',
2021-01-17T02:47:00.714449+00:00 app[web.1]: avatar_url: 'https://avatars2.githubusercontent.com/u/35227860?v=4',
2021-01-17T02:47:00.714450+00:00 app[web.1]: gravatar_id: '',
2021-01-17T02:47:00.714450+00:00 app[web.1]: url: 'https://api.github.com/users/Jean-Baptiste-Lasselle',
2021-01-17T02:47:00.714450+00:00 app[web.1]: html_url: 'https://github.com/Jean-Baptiste-Lasselle',
2021-01-17T02:47:00.714451+00:00 app[web.1]: followers_url: 'https://api.github.com/users/Jean-Baptiste-Lasselle/followers',
2021-01-17T02:47:00.714451+00:00 app[web.1]: following_url: 'https://api.github.com/users/Jean-Baptiste-Lasselle/following{/other_user}',
2021-01-17T02:47:00.714451+00:00 app[web.1]: gists_url: 'https://api.github.com/users/Jean-Baptiste-Lasselle/gists{/gist_id}',
2021-01-17T02:47:00.714452+00:00 app[web.1]: starred_url: 'https://api.github.com/users/Jean-Baptiste-Lasselle/starred{/owner}{/repo}',
2021-01-17T02:47:00.714452+00:00 app[web.1]: subscriptions_url: 'https://api.github.com/users/Jean-Baptiste-Lasselle/subscriptions',
2021-01-17T02:47:00.714453+00:00 app[web.1]: organizations_url: 'https://api.github.com/users/Jean-Baptiste-Lasselle/orgs',
2021-01-17T02:47:00.714453+00:00 app[web.1]: repos_url: 'https://api.github.com/users/Jean-Baptiste-Lasselle/repos',
2021-01-17T02:47:00.714453+00:00 app[web.1]: events_url: 'https://api.github.com/users/Jean-Baptiste-Lasselle/events{/privacy}',
2021-01-17T02:47:00.714453+00:00 app[web.1]: received_events_url: 'https://api.github.com/users/Jean-Baptiste-Lasselle/received_events',
2021-01-17T02:47:00.714453+00:00 app[web.1]: type: 'User',
2021-01-17T02:47:00.714454+00:00 app[web.1]: site_admin: false,
2021-01-17T02:47:00.714454+00:00 app[web.1]: name: null,
2021-01-17T02:47:00.714454+00:00 app[web.1]: company: 'Gravitee.io',
2021-01-17T02:47:00.714454+00:00 app[web.1]: blog: 'https://gravitee.io',
2021-01-17T02:47:00.714455+00:00 app[web.1]: location: 'Paris',
2021-01-17T02:47:00.714455+00:00 app[web.1]: email: null,
2021-01-17T02:47:00.714455+00:00 app[web.1]: hireable: null,
2021-01-17T02:47:00.714456+00:00 app[web.1]: bio: 'Fullstack architect, I totally fell into openstack years ago, then docker, n K8S. I focus on how you develop & operate software in a microservices world',
2021-01-17T02:47:00.714458+00:00 app[web.1]: twitter_username: null,
2021-01-17T02:47:00.714458+00:00 app[web.1]: public_repos: 198,
2021-01-17T02:47:00.714458+00:00 app[web.1]: public_gists: 4,
2021-01-17T02:47:00.714458+00:00 app[web.1]: followers: 10,
2021-01-17T02:47:00.714459+00:00 app[web.1]: following: 29,
2021-01-17T02:47:00.714459+00:00 app[web.1]: created_at: '2018-01-08T14:35:44Z',
2021-01-17T02:47:00.714459+00:00 app[web.1]: updated_at: '2021-01-15T12:52:29Z'
2021-01-17T02:47:00.714459+00:00 app[web.1]: }

Now retrieving all those informations from both Github and Circle CI API v1, will be a btach job, that will need the Github API Token obtained from the Github OAuth:

When will this bathc job launched ?
It will be launched every time the Github OAuth happens ;
If the batch job is already running when the GIthub OAuth happens, then the job is cancelled, and restarted from scratch: that's the only case, along wit the "sync force restart", where an event can cancel the batch job. For all other events, the batch job just continues if it already was running.
this batch job can be laucnhed on demand (from a click of the Authenticated user, weare in session, still access to the OAuth GIthub API Token) : that's the "sync" button. Whe the user clisks that button, then he is asked if he wants to force force restart current sync, if a sync already is running
this batch job will have to be transactional,and on a per user basis (when ghallagher gets to be multi user)
this batch job can be relaunched on a regular basis, by a sync launched by an RXJS repeat loop on client side (since done from client side, we should get the session,and therfore the Github API scoped Token from OAuth, again on serverside).

Ok so the Batch job will take a parameter : the Github API scoped Token. With this token, the batch will get the user name , and use it to persist job result to database :

this will have to be transactional, a first mongo database is used to store results as the job goes. When results are all confirmed, all changes will be pushed to database on which the Web UI is plugged
I will need reactivity from mongo database, to Web ui : see how to do that neat, sure with RXJS, get an example of that :
- https://rxdb.info/tutorials/server.html :tried this one , the angular example works great, but it seems highly technical, though seems worth the effort, and would be a first try with couchdb. Also is multi user with leader election!
- much more simple, rxjs observable, and websocket : https://blog.codewithdan.com/pushing-real-time-data-to-an-angular-service-using-web-sockets/ (quick adapt to svlete / sapper, and I don't have a need for leader election, since data will always be edited by a batch job, one batch job/ one user, no shared data between users, and readonly data on Web UI side)

even better, simpler example of pushed notifications from mongodb (perfect mongo) : https://dev.to/m10otb/real-time-notification-web-app-2ebn

Ok,n SO i'll o that one :

I need first to set up the svelte sapper, sothat it can get the notifications as well, same thing : https://dev.to/m10otb/real-time-notification-web-app-2ebn
then I will implement :
- the batch job,
- and the Endpoints to get all orgs, repos, git branch data (feded from Gitjhub API, filtered frfom OAuth scope, labelled with Circle CI API projects infos) from mongo DB
- after that I will create custom notifications to notiffy the svelte sapper web ui that the sync batch ahas finished
- and then the synced data gets pushed to the datastore in the svelte sapper app (same as ngrx sttore and react stroe, well you know) :
- https://svelte.dev/tutorial/writable-stores
- https://sapper.svelte.dev/docs#Stores

I will use exact same pattern, batch job and mongo db , to reactively update web ui for piplines execution state : it will be a sync, wtih a force sync button, Or... I don't know... actually no, it will be so much simpler to just limit API call to update execution state for each pipeline, or for a page of pipelines.... Yeah I just pull the pipeline states, I really don't care about seeing more than what I have on page, I don't want to conduct like large sets metrics calculations , it's always just about displaying the state of a few pipelines.

yes indeed : what i do ned to be extremely reactive, pushed and solid, etc... is the git repos list, the branch list as well

All in all :say I have a new Pipeline execution and it shows a new git branch , between two syncs :

Ok, I think the batch job,should be only about git repos and orgs, nothing more ?
this shoudl trigger a sync, to update database, but that sync
oh I know :
- what is important, is that I always have the most fresh git branch list for a given repo
- so I will need a resfresh button on every drop down ,to pull from github api directly the git branch list
- this refresh is rate limited
- and everytie this refresh happens, it compares the list it gets with the git branch list in database : if there are differences, new branch is directly added in database. with timestamp. batch job will delete this branch if and only if it has a new Github API result with more recent timestamp, in its batch result...?
- Ok, git branch informations will always be time stamped, not dealt with batch job, only directly from dropdown lists pull. Push and batch will only be for git repos and orgs
- all in all : git branch infos :
- always in pull mode, with rate limit (but sync button to force update with smaller ratelimit), and since user scoped, not even a timestamp,
- opening a drop down pulls

pipeline executions state are going to be pushed, but with rate limit like every 3 seconds (slow), and the push updates the database iff there are differences between persisted state, and new state

git repos and github orgs in push mode, BATCH 1 , batch 60 seconds, sync can be forced with 10 seconds rate limit, database persistence is on.
git branches of a repo in pull mode, rate limit 3 seconds, no database persistence, just logs, and later log analyzis to get more insights.
pipeline configuration (on a given branch) in pull mode, to construct the html from accordingly for params, rate limit 3 seconds, no database persistence, just logs, and later log analyzis to get more insights.
pipeline execution state in push mode BATCH 2, batch 3 seconds, sync cannot be forced, :
- no database persistence, just logs, and later log analyzis to get more insights.
- no database, but I keep in memory, the previous state of each pipeline, to compare is there is a change, and therefore If I need to push a data which ahs changed.
- batch is sleeping as long as no pipeline execution state page, has been opened.
- Batch goes to sleep when there is no more websotcket connection. See if can setup/teardown a websocket, based on the page url, and I if I needto know which pieplines are dsiplaed, to tel the batch for which pieplines I need the batch to push data for.
one source of truth for every information, never compare Circle CI and Github API infos : Actually, any comparison can be made completely independently of the ghallagher, by analyzing log aggregation to an ELK Stack.
mongo db as database,
all infos persisted in dataabse, with timestamp :
- every time I get data from Circle CI API, or Github API, I add to it a timestamp : the timestamp is created right after the response is received. then databse comparison is done, to decide persistence : is there a change, and if so, is my data fresher than in database ? (otherwise no persistence, and therefore as little database writes as possible. When fresher data persisted to database, if mode is push, then trigger the push through websocket. On Web UI Client side, subscription to websocket by RxJS to receive and update page

Conclusion of analysis : 2 main tasks

Task 1 : database persistence + Docekrize

Alright, So :

I need , according the below listed future endpoints implementations, deisgn the logical model of the mongo database (with Pipeline GUID as Primarykeys for pipeline triggers, for example=)
and i want the simplest possible pattern to easily carry out any persistence, using TypeORM
https://github.com/typeorm/typescript-example
https://typeorm.io/#/example-with-express : very simple in this example, the DAOfor the User entity, is the userRepository object in this example. K fine, let's roll with this
and a few specifics aboutmongo (eg for primary keys etc..) : https://typeorm.io/#/mongodb

Finally , Dockerize everything, I wan in the end a docker-compse and a helmchart (tested on K3D) with :

backend
frontend (svelte sapper ui)
mongodb (one docker container in docker-compose)
volumeto map the ./secret.json

later:

do the helmchart , with k3D and K8S secret for the ./secret.jsoncontaining the circle ci api token , alsoconsider how to use the secrethub inside the K8S cluster, todobetter,and set the secrethub credential as K8S secret
mongodb (in helm chart I want the usual official mongodb helm chart as a chart dependency, mongodb replicaset)

being able to persist to dataabse, shoudl be enough to implement all Endpoints in next Task :

Task 2 : create the needed Express Endpoint

Github API Client methods used by the batch and the Express Endpoints for pull :

Github API Token is passed as method parameter
list organizations
list repositories
list git branches for a given repository

For pull :

[ GET /backend/github/git-branches] : returns git branches of a given repo, in a given org, speicified with JSON Payload, logs, and no database persistence (require Github Authentication)
[ GET /backend/circleci/pipeline-config] : returns the YAML .circleci/config.yml of a given repo, in a given org, ona given git branch, specified with JSON Payload, logs, and no database persistence. On client side, either :
- YAML is parsed, parameters with their types, are infered, and HTML form generated to trigger pipeline
- or Pipeline configuration is displayed as raw YAML to show the pipeline config
modify the [ POST /backend/circleci/trigger-pipeline] , so that every time a pipeline execution is triggered, the value of the parameters is persisted to mongo database, with pipeline execution GUID (then all triggered pipeline execution can be listed
add a new [ GET /backend/circleci/pipeline-executions] returns the list of all pipeline executions, paginated with next page token , from a github org name , see https://circleci.com/docs/api/v2/#operation/listPipelines . pagination will be based on Circle CI Pagination :
- first we retrieve all pipeline execution states from Circle CI, with the next page token, using https://circleci.com/docs/api/v2/#operation/listPipelines ,
- and for each Pipeline GUID, it gets the parameters value passed when triggered, from mongodb database, to Pipeline GUID as primary key obviously
add a new [ GET /backend/circleci/pipeline-executions-of] returns the list of all pipeline executions of a given repo, paginated with next page token , from a github org name, and the repo name , see https://circleci.com/docs/api/v2/#operation/listPipelinesForProject . The data will be fetched from database and Circle CI API. pagination will be based on Circle CI Pagination :
- first we retrieve all pipeline execution states from Circle CI, with the next page token, using https://circleci.com/docs/api/v2/#operation/listPipelinesForProject ,
- and for each Pipeline GUID, it gets the parameters value passed when triggered, from mongodb database, to Pipeline GUID as primary key obviously
- and for each Pipeline GUID, it gets the parameters value passed when triggered, from mongodb database, to Pipeline GUID as primary key obviously
add a new [ GET /backend/circleci/pipeline-execution-state] which returns, from Pipeline GUID, the full detailed execution state of a given pipeline :
- not just the JSON returned by https://circleci.com/docs/api/v2/#operation/getPipelineById
- but also add detailed infos for all workflows of the pipeline execution state, using https://circleci.com/docs/api/v2/#operation/listWorkflowsByPipelineId and https://circleci.com/docs/api/v2/#operation/getWorkflowById
- this one will be in push mode, no database persistence, but logs for insights,
- this on will be used by the detail view of a pipeline in the Web UI, just like on Circle CI
- this endpoint will also use the database to retrieve the values passed to the parameters for this pipeline executions (to display it in detail page of a pipeeline execution)

Ok, so all in all, here what is new,is that I want to remember the values of the passed parameters : can Jenkins tell you that?I am not sure.

Task 3 : POC on push

Ok, So here I will need to experiment to websocket and push pattern :

deisgn how batch are implemented
how can i push to web ui trhough websocket

Future

Now, In this web ui ghallagher,I have a database with all pipeline parameters valeus for every Invocations :

those values are also logged, to be agregated int he ELK , just like the ORchestrator Oochestratinng the CICD Processes : Ok, that way "everything is sent to the ELK, for later deepanalysis
Ok, fine, Now what I'll also want be able to do from this UI, is to be able to get insights data from the ELK, not all, but some :
- if I want to generally analyze the system, I am not going to re-developpages "better thatn grafana", I'm jsut gonna go to grafana to see the reports.
- now, maybe there are some operations on the Monitoring system, that I'd liek to perform, frommy local ghallagher, someday : things that I cannot do with a grafana report. Maybe some helpers to create datasources ...or anythingI'll see. OR just perform a quick search across logs, deep dive into it. See also Grafana loki

gravitee-lab3 / svelte-workshop-backend