search

erdc / proteus

A computational methods and simulation toolkit
http://proteustoolkit.org
MIT License
88 stars 56 forks source link

minimize the proteus.linux2.yaml stack #687

Open cekees opened 6 years ago

cekees commented 6 years ago

By that I mean try commenting out as much of the stack as we can. The main dependencies we all know: petsc, hdf5, superlu, triangle, tetgen, but in some of the newer modules there are others like chrono, scipy, etc. I believe as long as you just comment out packages the stack rebuilds should go quickly and if you just remove proteus/linux2 the proteus modules won't rebuild either, so testing the stack by hand shouldn't take all that long and will shave some time off the downloads of the binaries. On the other hand the really big packages are probably petsc and hdf5, which we can't do anything about.. By that I mean try commenting out as much of the stack as we can. The main dependencies we all know: petsc, hdf5, superlu, triangle, tetgen, but in some of the newer modules there are others like chrono, scipy, etc. I believe as long as you just comment out packages the stack rebuilds should go quickly and if you just remove proteus/linux2 the proteus modules won't rebuild either, so testing the stack by hand shouldn't take all that long and will shave some time off the downloads of the binaries. On the other hand the really big packages are probably petsc and hdf5, which we can't do anything about.

zhang-alvin commented 6 years ago

  1. From a quick look at the listed packages, I don't see any exposure in the code to julia or cgal. The other packages that might not generally be used are packages for plotting like openjpeg, matplotlib, matlab and for manipulating certain data formats like GRIB ( grib_api).

  2. Some of the packages are needed in the C++ code (e.g. scorec), which would require more than just deleting proteus/linux2 to reset the build. So unless it's too expensive, I'd rather just let Travis handle testing the stack profile.

  3. It might be useful to add a note next to each package indicating how they are used. That way, we can have a better idea of when packages become obsolete.

cekees commented 6 years ago

On 1) yes, and we can use pip install for matplotlib if it's not being used for any tests. On 2) hashdist is going to ensure that scorec dependencies are met, so unless there are undeclared dependencies in the scorec.yaml file deleting proteus/linux2 will regenerate the stack properly. hashdist uses a sort of build jail to try to prevent linking to external dependencies accidentally. It's fine to let travis do the checking if you want to, but I was trying to save you time. Good idea on 3) about adding a note. You could just put a generic "optional package" in the ones you comment out for now.

zhang-alvin commented 6 years ago

1) It turns out matplotlib is used by the testing suite. 2) My concern was really that if a C++ extension/module was already compiled, the .so and .a files are not stored in proteus/linux2 so even if I commented out some packages and deleted proteus/linux2, it wouldn't prompt a rebuild of the C++ libraries.

I've commented out a good number of packages that can be seen in the following erdc/hashstack to hashdist/hashstack PR: hashdist/hashstack#997

If anyone else has any other packages that they think can be removed, let me know.

The following is the Travis test associated with the test stack: https://travis-ci.org/erdc/proteus/jobs/337657934