Building Broken

[bilalshaikh42]

Commit e9fdc6193a3e8669abfc04f7eebaefff8b8b43e2 is preventing vcell and the biosimulators docker images from building.

[moraru]

@bilalshaikh42 did you update vcell-solvers code (separate repo) in the docker image? it needs v.0.0.35 (https://github.com/virtualcell/vcell-solvers/commit/4834104f011463a75bd511adfca0991a0f284e19)

[bilalshaikh42]

I am using the same build process as the ci for the repo. The ci itself is failing

[moraru]

Have not been involved in the CI but I noticed that the offending commit you mentioned happens to be one of the rare code changes that require updating a dependency (which sits in a different repo, vcell-solvers) to a newer version.

Ion

Sent from my iPhone, please excuse any typos.

On Jun 28, 2021, at 18:56, Bilal Shaikh @.***> wrote:

 Attention: This is an external email. Use caution responding, opening attachments or clicking on links.

I am using the same build process as the ci for the repo. The ci itself is failing

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/virtualcell/vcell/issues/72*issuecomment-870098844__;Iw!!N0rdg9Wr!_ADm4Pf4ZmTJI_Ar-rtDDRHayw_RlmqWGYhINNbUm5obrcK9Tu6-bo1Hx_HQ5Q$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ABYODNT2ERNUY4MGDYAPJPLTVD4WFANCNFSM47H3R6IA__;!!N0rdg9Wr!_ADm4Pf4ZmTJI_Ar-rtDDRHayw_RlmqWGYhINNbUm5obrcK9Tu6-bo2GzboBFw$.

[moraru]

7.4.0.10 did build (it was deployed to alpha) is right before that. Frank Morgan (who made that commit) happens to have taken over the duties of CI/deployment from Akhil, I'll let him know to fix it. @jonrkarr can you please add Frank to biosimulations (vcfrmgit is his handle)

[moraru]

@bilalshaikh42 is the branch that you you made yesterday that ends in tag 7.4.0.12 building OK and what is currently deployed on biosimulations?

[bilalshaikh42]

The image is being built directly from this repo, so there is no need to add to the biosimulations org.

The tag is of a branch in which I removed the commits that were causing build errors, so the history is a bit diverged from the main branch. I have not been able to get it to pass all the tests and get uploaded to biosimulations yet

On Tue, Jun 29, 2021 at 10:03 AM moraru @.***> wrote:

@bilalshaikh42 https://github.com/bilalshaikh42 is the branch that you you made yesterday that ends in tag 7.4.0.12 building OK and what is currently deployed on biosimulations?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/virtualcell/vcell/issues/72#issuecomment-870629489, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHX4FIB232PHTORHJRRTNODTVHHERANCNFSM47H3R6IA .

[moraru]

The CI build is fixed now, @bilalshaikh42 can you please merge back with all removed commits and all your other changes

[bilalshaikh42]

Done

[bilalshaikh42]

New releases from here should trigger the biosimulators image to build and get pushed

[bilalshaikh42]

The image is building, but running it fails with the following error

org.vcell.util.exe.ExecutableException: Unexpected error: Cannot run program "/usr/local/app/vcell/installDir/localsolvers/linux64/SundialsSolverStandalone_x64" (in directory "/tmp/out/subdir/BIOMD0000000912_sim.sedml"): error=2, No such file or directory

@vcfrmgit Could you take a look at the dockerfile to see if there is a missing step?

[jonrkarr]

I think the issue is the location where VCell is expecting SED-ML files because one of the test cases does pass. It doesn't look to me that there's an issue with the Docker image specifically.

I recommend installing the test suite locally. This enables faster iteration than the GitHub action, including the options to (a) run individual test cases and (b) save the OMEX files that the test suite generates for testing VCell so they can be inspected. https://github.com/biosimulators/Biosimulators_test_suite

[bilalshaikh42]

Im not sure about that. The missing file in the error is one of the solvers needed by vcell

On Tue, Jun 29, 2021 at 1:53 PM Jonathan Karr @.***> wrote:

I think the issue is the location where VCell is expecting SED-ML files because one of the test cases does pass. It doesn't look to me that there's an issue with the Docker image specifically.

I recommend installing the test suite locally. This enables faster iteration than the GitHub action, including the options to (a) run individual test cases and (b) save the OMEX files that the test suite generates for testing VCell so they can be inspected. https://github.com/biosimulators/Biosimulators_test_suite

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/virtualcell/vcell/issues/72#issuecomment-870797622, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHX4FIHEMN52DJF2XWDCFL3TVICBBANCNFSM47H3R6IA .

[jonrkarr]

Could be. I wasn't sure what the missing message refers to -- solver or simulation.

[moraru]

@bilalshaikh42 is right - the only successful executed archive is the Ciliberto - Fehlberg but that one specifies RK5 solver which is one of the few VCell Java solvers; it appears that CVODE and all binary solvers are missing from the image

[moraru]

As I mentioned, all the non-Java solvers are in a different repository. They need to be pulled into the image. In local, maven does that for you. Below is the relevant section from the vcell-core pom.xml.

                            <execution>
                                <id>install-solvers-linux64</id>
                                <phase>generate-test-resources</phase>
                                <goals>
                                    <goal>wget</goal>
                                </goals>
                                <configuration>
                                    <url>https://github.com/virtualcell/vcell-solvers/releases/download/v0.0.35/linux64.tgz</url>
                                    <unpack>true</unpack>
                                    <outputDirectory>${project.build.directory}/../../localsolvers/linux64</outputDirectory>
                                </configuration>
                            </execution>

[bilalshaikh42]

I see the issue. The solvers are being copied from this repo. Do you know what the path ${project.build.directory}/../../localsolvers/linux64 will resolve to relative to the root of the repo?

[moraru]

I believe that https://github.com/virtualcell/vcell/tree/master/localsolvers/linux64 is what you are asking about (which is empty since non-Java solvers are now developed in the other repo and the latest release needs to be copied over at build time)

(and from the docker image perspective it is /usr/local/app/vcell/installDir/localsolvers/linux64/)

[bilalshaikh42]

Yes, so currently, the docker image is using maven to build the app, and then copy over the needed files. To this, I need it to also copy the over the solvers that were automatically downloaded by Maven. So essentially, I need to know where maven is saving the solvers during the build process. The maven install command is run at the root of the repo. The output path provided in the pom.xml (${project.build.directory}/../../localsolvers/linux64) is not clear to me, as that seems to the now unused folder you mentioned above.

I realize that I could use the docker build to download and place the solvers in the correct place manually, but I would rather have the image use the maven files to prevent divergence between biosimulators image and the main build process.

[bilalshaikh42]

COPY --from=build /app/localsolvers /usr/local/app/vcell/installDir/localsolvers Here is what I am doing currently. the "/app" directory is the root of the repo. So the localsolvers folder is copied over into the docker image. I need to change this from the localsolvers folder to the folder where maven is downloading the new solvers

[vcfrmgit]

Hi All,

 The solvers are built by travis, appveyor and dockerhub when a 'tag' is made on the github virtualvcell/vcell-solvers repo, when they finish they upload the results to github, below is the latest Linux solvers build archive:

 https://github.com/virtualcell/vcell-solvers/releases/download/v0.0.35/linux64.tgz.

When VCell builds (https://github.com/virtualcell/vcell) maven pulls the solvers it will package into the clients for local solver runs from the above url (win, mac, linux).

Thanks,

Frank Morgan

@.**@.>

---

From: Bilal Shaikh @.***> Sent: Tuesday, June 29, 2021 6:32 PM To: virtualcell/vcell Cc: Morgan,Frank; Mention Subject: Re: [virtualcell/vcell] Building Broken (#72)

Attention: This is an external email. Use caution responding, opening attachments or clicking on links.

COPY --from=build /app/localsolvers /usr/local/app/vcell/installDir/localsolvers Here is what I am doing currently. the "/app" directory is the root of the repo. So the localsolvers folder is copied over into the docker image. I need to change this from the localsolvers folder to the folder where maven is downloading the new solvers

- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/virtualcell/vcell/issues/72*issuecomment-870962479__;Iw!!N0rdg9Wr!8CI3hGo3NdwZ3cYSbOrFEooFsApKmB0VXcggwIjZjIJiWyc-Ubuo_rYDsm4y$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AG7BGMLWWS75EMD3HVJMCFDTVJCXPANCNFSM47H3R6IA__;!!N0rdg9Wr!8CI3hGo3NdwZ3cYSbOrFEooFsApKmB0VXcggwIjZjIJiWyc-Ubuo_pyNcfC-$.

[moraru]

Yes, so currently, the docker image is using maven to build the app, and then copy over the needed files. To this, I need it to also copy the over the solvers that were automatically downloaded by Maven. So essentially, I need to know where maven is saving the solvers during the build process. The maven install command is run at the root of the repo. The output path provided in the pom.xml (${project.build.directory}/../../localsolvers/linux64) is not clear to me, as that seems to the now unused folder you mentioned above.

I realize that I could use the docker build to download and place the solvers in the correct place manually, but I would rather have the image use the maven files to prevent divergence between biosimulators image and the main build process.

Yes, it points to that "unused' folder. It is not populated in the main vcell repo, but after maven does its thing, it should be populated with the binaries pulled from the latest build of the other repo. So you should find them there.

[vcfrmgit]

Hi All,

Log into the host where you built the docker image

"ssh ***@***.***Host"

Find docker Image id:

"docker images" (look for your docker image, get number under "Image ID")

Run your docker image and log into it with bash shell:

 "docker run -it --entrypoint /bin/bash ImageID"

If above not work try - "docker run -it ImageID /bin/bash"

if error bash not exist then use /bin/sh instead

Search entire container filesystem for the location of an expected Linux solver:

"find / -name 'FiniteVolume_x64' -print"

For Example, in the VCell docker container "vcell-clientgen" that I use to package local solvers the full pathname found was /vcellclient/localsolvers/linux64/FiniteVolume_x64.

The vcell-clientgen container came from an image generated by a dockerfile that has an entry:

"COPY ./localsolvers /vcellclient/localsolvers" (docker copy from ./localsolvers on your disk into dockerimage at /vcellclient/localsolvers).

The "./localSolvers" was generated by maven build command as Ion mentioned in a previous email

Your Docker file (a human readable text file) used to create the dockerimage that is run to create a dockercontainer has more info about what and where files are stored.

Thanks,

Frank Morgan

---

From: moraru @.***> Sent: Tuesday, June 29, 2021 7:15 PM To: virtualcell/vcell Cc: Morgan,Frank; Mention Subject: Re: [virtualcell/vcell] Building Broken (#72)

Attention: This is an external email. Use caution responding, opening attachments or clicking on links.

Yes, so currently, the docker image is using maven to build the app, and then copy over the needed files. To this, I need it to also copy the over the solvers that were automatically downloaded by Maven. So essentially, I need to know where maven is saving the solvers during the build process. The maven install command is run at the root of the repo. The output path provided in the pom.xml (${project.build.directory}/../../localsolvers/linux64) is not clear to me, as that seems to the now unused folder you mentioned above.

I realize that I could use the docker build to download and place the solvers in the correct place manually, but I would rather have the image use the maven files to prevent divergence between biosimulators image and the main build process.

Yes, it points to that "unused' folder. It is not populated in the main vcell repo, but after maven does its thing, it should be populated with the binaries pulled from the latest build of the other repo. So you should find them there.

- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/virtualcell/vcell/issues/72*issuecomment-870978951__;Iw!!N0rdg9Wr!-yYZk74YzcHvWMAUY9Q74o9CKS6-zd734bMC5rx0tRhDvBnE7udyNkrIYdZZ$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AG7BGMPCUQ3MICMNYK6HZXLTVJHX7ANCNFSM47H3R6IA__;!!N0rdg9Wr!-yYZk74YzcHvWMAUY9Q74o9CKS6-zd734bMC5rx0tRhDvBnE7udyNoBMDFvx$.

[bilalshaikh42]

I am running into the following issue

java.nio.file.NoSuchFileException: ./.vcell/simdata/temp/300991193_link/SundialsSolverStandalone_x64

@moraru this seems to be an issue with where vcell is looking for the solver (caused by the cli code, not the main app). I am not familiar with the implementation and structure of the vcell repo, so will have to defer to the vcell team. Could someone from the team work on this? The dockerfile is quite easy to follow

[jonrkarr]

@moraru @vcfrmgit We're trying to get VCell working, but its hard for us to navigate the VCell code. Please use the test suite to check that the VCell Docker image works.

In the meantime, I think we will have to disable VCell in RunBioSimulations so Bilal can deploy a new version where every selectable simulation tool and example simulation run works.

[moraru]

@jonrkarr @bilalshaikh42 the last image that Akhil build before leaving was passing all tests - why don't we leave that one for now until we figure out why containers no longer work after @bilalshaikh42 tried to automate it to go directly from the main vcell repo?

[moraru]

on the other hand, perhaps @gmarupilla can take a quick look and be able to identify what is missing with the current build procedure, since he was doing all the builds before without any issues

[jonrkarr]

We needed to introduce a few minor changes to BioSimulators-utils to work around some quirks of the HSDS that Bilal is using for simulation results. This should be orthogonal to VCell. The image should just need to be rebuilt. I think Bilal was having trouble figuring out how to rebuild the VCell image.

[moraru]

seems like the easiest way for now is to simply build an image using our routine alpha deployment and have Bilal just manually copy that image to biosimulations - @vcfrmgit can you please deploy the latest master (I think it should be tagged as 7.4.0.14) and send a link to @bilalshaikh42 to download the singularity image?

[bilalshaikh42]

Resolved