Supporting local and git-based builds of the rosetta_omp conda package

[weitzner]

The initial version of the rosetta_omp conda build scripts required access to the Rosetta/main GitHub repository, which complicates using these tools with the released code. This PR adds support for producing conda builds from the released code (rosetta 3.9).

In addition to ./docker_build.sh rosetta_omp [CONDA CHANNEL PATH] [GITHUB TOKEN FILE] the build tools can now also be invoked with ./docker_build.sh rosetta_local [CONDA CHANNEL PATH] [GITHUB TOKEN FILE]

While the token isn't strictly required in this case, it is simpler to main this interface. The script assumes the rosetta tarball is extracted in the same directory as rifdock-recipes (i.e., if you can issue ./docker_build.sh, the path to the appropriate version of rosetta is ../rosetta_src_2018.09.60072_bundle).

This addresses #5

[mchaker]

Thank you so much for this!

[weitzner]

Ok, @mchaker, I think this is ready for you to test. Want to give it a go and let me know if you run into any issues?

[mchaker]

Testing this today @weitzner

[mchaker]

Hit this error

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
can't find file to patch at input line 5
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff --git a/source/cmake/build/build.settings.cmake b/source/cmake/build/build.settings.cmake
|index 1f73699632b..b0efe0745b3 100644
|--- a/source/cmake/build/build.settings.cmake
|+++ b/source/cmake/build/build.settings.cmake
--------------------------
No file to patch.  Skipping patch.
1 out of 1 hunk ignored
can't find file to patch at input line 17
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff --git a/source/cmake/build_cxx11_omp/CMakeLists.txt b/source/cmake/build_cxx11_omp/CMakeLists.txt
|index 23082655a18..622c1845ae5 100644
|--- a/source/cmake/build_cxx11_omp/CMakeLists.txt
|+++ b/source/cmake/build_cxx11_omp/CMakeLists.txt
--------------------------
No file to patch.  Skipping patch.
3 out of 3 hunks ignored
Copying /home/conda/rosetta to /home/conda/build/rosetta_omp_1650127052132/work/
Applying patch: /home/conda/root/patch-rosetta-for-conda-build.diff
Patch level ambiguous, selecting least deep
Applying patch: /home/conda/root/patch-rosetta-for-conda-build.diff with args:
['-Np0', '-i', '/tmp/tmpxmz0kpih/patch-rosetta-for-conda-build.diff.native', '--binary']
Warning: failed to download source.  If building, will try again after downloading recipe dependencies.
Error was: 
Command '['/usr/bin/patch', '--no-backup-if-mismatch', '--batch', '-Np0', '-i', '/tmp/tmpxmz0kpih/patch-rosetta-for-conda-build.diff.native', '--binary', '--dry-run']' returned non-zero exit status 1.
Failed to download or patch source. Please see build log for info.

[mchaker]

Oh, I had the bin version of rosettta downloaded, not the src version. oops. Retrying with src

[mchaker]

Ok, still hit an error. Different this time:

    cmd(state, arg)
  File "/opt/conda/lib/python3.7/site-packages/conda/instructions.py", line 67, in PROGRESSIVEFETCHEXTRACT_CMD

    progressive_fetch_extract.execute()
  File "/opt/conda/lib/python3.7/site-packages/conda/core/package_cache_data.py", line 620, in execute
    raise CondaMultiError(exceptions)
conda.CondaMultiError: Signal interrupt SIGTERM
Signal interrupt SIGTERM

{ 
  "rosetta_omp-2018.09.60072-h75f3eb4_0": {
    "recipe": {
      "c_compiler": "gcc",
      "c_compiler_version": "7",
      "channel_targets": "conda-forge main",
      "cxx_compiler": "gxx",
      "cxx_compiler_version": "7",
      "libgomp": "7",
      "target_platform": "linux-64"
    }
  }
}

####################################################################################
Resource usage summary:

Total time: 0:28:49.2
CPU usage: sys=0:19:37.7, user=5:11:30.2
Maximum memory usage observed: 25.0G
Total disk usage observed (not including envs): 2.9M

[mchaker]

A warning hit along the way to that error:

WARNING (rosetta_omp,lib/libprotocols_f.5.so): $RPATH/libm.so.6 not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
   INFO (rosetta_omp,lib/libprotocols_f.5.so): Needed DSO lib/libgcc_s.so.1 found in pkgs/main::libgcc-ng-9.1.0-hdf63c60_0
WARNING (rosetta_omp,lib/libprotocols_f.5.so): $RPATH/libpthread.so.0 not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?

There are other warnings like this, ending in "is this binary repackaging?"

[mchaker]

I ran the build command again and see these messages at the end of the build output:

INFO:conda_build.utils:shutil.move(work)=/home/conda/build/rosetta_omp_1650131959887/work, dest=/home/conda/build/rosetta_omp_1650131959887/work_moved_rosetta_omp-2018.09.60072-h75f3eb4_0_linux-64)
Collecting package metadata: ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /home/conda/build/rosetta_omp_1650131959887/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold

The following NEW packages will be INSTALLED:

    _libgcc_mutex: 0.1-main                 defaults                
    _openmp_mutex: 4.5-1_gnu                defaults                
    libgcc-ng:     9.1.0-hdf63c60_0         defaults                
    libgomp:       9.3.0-h5101ec6_17        defaults                
    libstdcxx-ng:  9.1.0-hdf63c60_0         defaults                
    rosetta_omp:   2018.09.60072-h75f3eb4_0 file:///home/conda/build
    zlib:          1.2.11-h7f8727e_4        defaults                

Preparing transaction: ...working... done
Verifying transaction: ...working... done
ERROR:conda.stderr:
ClobberWarning: This transaction has incompatible packages due to a shared path.
  packages: defaults::libgcc-ng-9.1.0-hdf63c60_0, defaults::libgomp-9.3.0-h5101ec6_17
  path: 'lib/libgomp.so'

ClobberWarning: This transaction has incompatible packages due to a shared path.
  packages: defaults::libgcc-ng-9.1.0-hdf63c60_0, defaults::_openmp_mutex-4.5-1_gnu
  path: 'lib/libgomp.so.1'

ClobberWarning: This transaction has incompatible packages due to a shared path.
  packages: defaults::libgcc-ng-9.1.0-hdf63c60_0, defaults::libgomp-9.3.0-h5101ec6_17
  path: 'lib/libgomp.so.1.0.0'

Executing transaction: ...working... ERROR:conda.stderr:
ClobberWarning: Conda was asked to clobber an existing path.
  source path: /opt/conda/pkgs/libgomp-9.3.0-h5101ec6_17/lib/libgomp.so
  target path: /home/conda/build/rosetta_omp_1650131959887/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/libgomp.so

ERROR:conda.stderr:
ClobberWarning: Conda was asked to clobber an existing path.
  source path: /opt/conda/pkgs/libgomp-9.3.0-h5101ec6_17/lib/libgomp.so.1.0.0
  target path: /home/conda/build/rosetta_omp_1650131959887/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/libgomp.so.1.0.0

ERROR:conda.stderr:
ClobberWarning: Conda was asked to clobber an existing path.
  source path: /opt/conda/pkgs/_openmp_mutex-4.5-1_gnu/lib/libgomp.so.1
  target path: /home/conda/build/rosetta_omp_1650131959887/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/libgomp.so.1

done
+ rosetta_scripts
+ exit 0
export PREFIX=/home/conda/build/rosetta_omp_1650131959887/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold
export SRC_DIR=/home/conda/build/rosetta_omp_1650131959887/test_tmp
core.init: Rosetta version: rosetta.source.release-171 r171 2018.09+release.333d9969977 333d99699777cd6a50d1c1736bacd689f8f1d1df https://www.rosettacommons.org 2018-02-27T20:08:15.525183
core.init: command: rosetta_scripts
core.init: 'RNG device' seed mode, using '/dev/urandom', seed=736232896 seed_offset=0 real_seed=736232896
core.init.random: RandomGenerator:init: Normal mode, seed=736232896 RG_type=mt19937
core.init: Resolved executable path: $PREFIX/bin/rosetta_scripts
core.init: Looking for database based on location of executable: $PREFIX/bin/../database/
core.init: 
core.init: USEFUL TIP: Type -help to get the options for this Rosetta executable.
core.init: 
protocols.RosettaScripts.util: No XML file was specified with the "-parser:protocol <filename>" commandline option.  In order for RosettaScripts to do something, it must be provided with a script.
protocols.RosettaScripts.util: The following is an empty (template) RosettaScripts XML file:

<ROSETTASCRIPTS>
        <SCOREFXNS>
        </SCOREFXNS>
        <RESIDUE_SELECTORS>
        </RESIDUE_SELECTORS>
        <TASKOPERATIONS>
        </TASKOPERATIONS>
        <FILTERS>
        </FILTERS>
        <MOVERS>
        </MOVERS>
        <APPLY_TO_POSE>
        </APPLY_TO_POSE>
        <PROTOCOLS>
        </PROTOCOLS>
        <OUTPUT />
</ROSETTASCRIPTS>

At any point in a script, you can include text from another file using <xi:include href="filename.xml" />.
protocols.RosettaScripts.util: Variable substituion is possible from the commandline using the -"parser:script_vars varname=value" flag.  Any string of the pattern "%%varname%%" will be replaced with "value" in the script.
protocols.RosettaScripts.util: 
protocols.RosettaScripts.util: The rosetta_scripts application will now exit.

Resource usage statistics from testing rosetta_omp:
   Process count: 1
   CPU time: Sys=0:00:00.0, User=-
   Memory: 880.0K
   Disk usage: 12B
   Time elapsed: 0:00:02.1

TEST END: /home/conda/build/linux-64/rosetta_omp-2018.09.60072-h75f3eb4_0.tar.bz2
Renaming work directory '/home/conda/build/rosetta_omp_1650131959887/work' to '/home/conda/build/rosetta_omp_1650131959887/work_moved_rosetta_omp-2018.09.60072-h75f3eb4_0_linux-64_main_build_loop'
shutil.move(work)=/home/conda/build/rosetta_omp_1650131959887/work, dest=/home/conda/build/rosetta_omp_1650131959887/work_moved_rosetta_omp-2018.09.60072-h75f3eb4_0_linux-64_main_build_loop)
INFO:conda_build.utils:Renaming work directory '/home/conda/build/rosetta_omp_1650131959887/work' to '/home/conda/build/rosetta_omp_1650131959887/work_moved_rosetta_omp-2018.09.60072-h75f3eb4_0_linux-64_main_build_loop'
INFO:conda_build.utils:shutil.move(work)=/home/conda/build/rosetta_omp_1650131959887/work, dest=/home/conda/build/rosetta_omp_1650131959887/work_moved_rosetta_omp-2018.09.60072-h75f3eb4_0_linux-64_main_build_loop)
# Automatic uploading is disabled
# If you want to upload package(s) to anaconda.org later, type:

# To have conda build upload to anaconda.org automatically, use
# conda config --set anaconda_upload yes
anaconda upload \
    /home/conda/build/linux-64/rosetta_omp-2018.09.60072-h75f3eb4_0.tar.bz2
anaconda_upload is not set.  Not uploading wheels: []

INFO :: The inputs making up the hashes for the built packages are as follows:
{
  "rosetta_omp-2018.09.60072-h75f3eb4_0": {
    "recipe": {
      "c_compiler": "gcc",
      "c_compiler_version": "7",
      "channel_targets": "conda-forge main",
      "cxx_compiler": "gxx",
      "cxx_compiler_version": "7",
      "libgomp": "7",
      "target_platform": "linux-64"
    }
  }
}

####################################################################################
Resource usage summary:

Total time: 0:27:09.0
CPU usage: sys=0:19:40.8, user=5:10:38.4
Maximum memory usage observed: 26.1G
Total disk usage observed (not including envs): 2.9M

[mchaker]

@weitzner I tried running ./docker_build.sh rifdock ... and got this error:

Executing transaction: ...working... done
Cloning into bare repository '/home/conda/build/git_cache/github.com/rifdock/rifdock.git'...
warning: redirecting to https://github.com/rifdock/rifdock.git/
Cloning into '/home/conda/build/rifdock_1650135294570/work'...
error: pathspec '9ece4162b790a2e1f386638a28f74206205e352' did not match any file(s) known to git
checkout: '9ece4162b790a2e1f386638a28f74206205e352'
Warning: failed to download source.  If building, will try again after downloading recipe dependencies.
Error was: 
Command '['/opt/conda/bin/git', 'checkout', '9ece4162b790a2e1f386638a28f74206205e352']' returned non-zero exit status 1.
Failed to download or patch source. Please see build log for info.

[mchaker]

@weitzner also, for docker_build.sh, does it handle the rifdock argument at all? I don't see it as part of the if-block in the script. (Maybe I'm missing something...)

EDIT: It probably does handle it by passing it as argument $1 somewhere, hmmm... still not sure how to make it work 😅

[weitzner]

Your output above shows a successful build! To build rifdock, use ./docker_build.sh rifdock [CONDA CHANNEL PATH] [GITHUB TOKEN FILE]

This sets the ROOT variable on line 17, which is then mounted to the container on line 50 of the build script.

[mchaker]

I see... so the docker_build.sh rifdock output in comment https://github.com/weitzner/rifdock-recipes/pull/6#issuecomment-1100735626 is a successful build? It seems like a git error/failure

[weitzner]

No, sorry about that - the rosetta build was successful. I don't know why that revision of rifdock would fail, but I have updated to the most recent revision on the main branch nonetheless. Give a shot

[mchaker]

@weitzner does this mean the docker_build.sh rifdock build succeeded?:

TEST END: /home/conda/build/linux-64/rifdock-0.9.9ece4162b79-h75f3eb4_0.tar.bz2
Renaming work directory '/home/conda/build/rifdock_1650446711241/work' to '/home/conda/build/rifdock_1650446711241/work_moved_rifdock-0.9.9ece4162b79-h75f3eb4_0_linux-64_main_build_loop'
shutil.move(work)=/home/conda/build/rifdock_1650446711241/work, dest=/home/conda/build/rifdock_1650446711241/work_moved_rifdock-0.9.9ece4162b79-h75f3eb4_0_linux-64_main_build_loop)
INFO:conda_build.utils:shutil.move(work)=/home/conda/build/rifdock_1650446711241/work, dest=/home/conda/build/rifdock_1650446711241/work_moved_rifdock-0.9.9ece4162b79-h75f3eb4_0_linux-64_main_build_loop)
# Automatic uploading is disabled
# If you want to upload package(s) to anaconda.org later, type:

# To have conda build upload to anaconda.org automatically, use
# conda config --set anaconda_upload yes
anaconda upload \
    /home/conda/build/linux-64/rifdock-0.9.9ece4162b79-h75f3eb4_0.tar.bz2
anaconda_upload is not set.  Not uploading wheels: []

INFO :: The inputs making up the hashes for the built packages are as follows:
{
  "rifdock-0.9.9ece4162b79-h75f3eb4_0": {
    "recipe": {
      "c_compiler": "gcc",
      "c_compiler_version": "7",
      "channel_targets": "conda-forge main",
      "cxx_compiler": "gxx",
      "cxx_compiler_version": "7",
      "libgomp": "7",
      "target_platform": "linux-64"
    }
  }
}

####################################################################################
Resource usage summary:

Total time: 0:14:03.2
CPU usage: sys=0:00:17.9, user=0:06:58.1
Maximum memory usage observed: 7.2G
Total disk usage observed (not including envs): 98.6K

####################################################################################
Source and build intermediates have been left in /home/conda/build.
There are currently 1 accumulated.
To remove them, you can run the ```conda build purge``` command

[weitzner]

Yes! Congratulations!

[mchaker]

Excellent! I'll continue down the instructions. This PR seems ready to be merged :)