EOSC-synergy / SQAaaS

6 stars 1 forks source link

Adding multiple services to new pipeline #28

Open ldelcano opened 3 years ago

ldelcano commented 3 years ago

I am trying to create a new pipeline for Scipion service and I would like to build two docker images: master and worker. I am adding three services: one for hadolint, one for master and one for worker but then I am not sure what to do with the Criteria. Should I have one single hadolint service with two commands to build master and worker or should I have two hadolint services one to build each docker? I have tried first option but it seems to me that the pipeline is not correct and mix things.

orviz commented 2 years ago

@ldelcano it makes sense to have just one service (where hadolint tool is available) for running the commands. If you still find any issue, please add here the pipeline files being produced by SQAaaS (config.yml, docker-compose.yml, Jenkinsfile)

ldelcano commented 2 years ago

I am then creating a Pipeline with a single service for each docker file and two hadolint commands on the same criteria and I keep getting this error: Parents Bind credentials to variables : Body : Start Error hudson.AbortException: script returned exit code 1 at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.handleExit(DurableTaskStep.java:659) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:605) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:549) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

I do not have a clue on how to solve it. I have tried several times, with one service, with two, changing names... I have chosen to push to dockerhub using EOSC synergy credentials. I coud try not to push in case it has to do with that.

orviz commented 2 years ago

Hi @ldelcano could you please provide the link to the jenkins job that correspond to these logs?

ldelcano commented 2 years ago

Hi @orviz here you have https://jenkins.eosc-synergy.eu/job/eosc-synergy-org/job/ScipionCloud-master.sqaaas/job/master/3/execution/node/32/

by the way I have problems understanding when I am creating a new pipeline and when not cause I cannot see a list of what I have (not sure if it gets saved or what).

orviz commented 2 years ago

Thanks @ldelcano the job you point is doing fine, it fails because hadolint finds some things to improve in the Dockerfile, and thus this tool returns an "exit code 1". If you have arrived to the logs via the "Pipeline Steps", to check the output of the hadolint tool, click instead in the console logo on the right hand side: image

or click here

Regarding the last question, you are creating a new pipeline every time you modify something. Whenever you get to the last step in the composition, you can check the contents of the files created in the "JePL files" and/or download them. This is the usual workflow.

Only when you want to check the execution of the pipeline (and you click "Try Out" for that) the SQAaaS creates a repo in GitHub so that Jenkins can detect and run it automatically. But this is part of the operational part of SQAaaS, thus not relevant for the common use cases.

ldelcano commented 2 years ago

Thanks @orviz I had check the console log but since I found only warnings I did not think this will stop the execution. But now that you mention this I think in the tutorial video you talk about this behavior. Sorry for not remembering it! I will try to continue.

ldelcano commented 2 years ago

Hi again, I have ignored all warnings (some might be reviewed later but just to keep moving) but it does not finishes. It seems to be a timeout when pushing to eosc-synergy github but I am not sure what the problem is:

https://jenkins.eosc-synergy.eu/job/eosc-synergy-org/job/ScipionCloud-worker.sqaaas/job/master/8/flowGraphTable/

Image is very big, is there a time limit to be pushed?

orviz commented 2 years ago

not sure about the limit, the truth is that Docker Hub is adding more a more constraints lately for the free accounts.. If the master-image is lighter, could you have a try? Besides if you have access to a different registry (other than Docker Hub), it could be also a valuable test

ldelcano commented 2 years ago

I have both images in my personal account in Dockerhub and I can push them without problem (they take long time but do not timeout). Could it be something related to the jenkins environment?

In any case I will rerun it not pushing to dockerhub. I will also try master image which should be lighter since it relies on worker image.

ldelcano commented 2 years ago

Hi @orviz I just tried to run a pipeline for master but it timeout pulling worker docker image (master image relies on worker). Then I tried to run the worker pipeline without pushing to dockerhub and it worked but I have a doubt. When I look here https://jenkins.eosc-synergy.eu/job/eosc-synergy-org/ I can see all my pipelines but it is a mess. Is it possible to delete them and start from scratch? In the eosc-synergy sqaaas portal to create pipelines I never know which pipeline I am creating. For instance today I started to work on ScipionCloud-master one and it failed, then I wanted to work on ScipionCloud-worker and I believe I renamed it (hitting back button) but then it appears under ScipionCloud-master name!!! How can I have a new clean pipeline that works? And I think timeouts should be reviewed cause it must have to do with your portal, I never gets timeout pulling or pushing from dockerhub even if my images are huge. Thanks