Closed aday00 closed 8 years ago
So per the above I will compare the RPM list to what you list. We do not make modules for RPM loaded items as RPM items are in the default path. I will then compare to the list of modules.
Historically on this cluster most Python needs are handled with personal or group Anaconda trees.
So from your request to what is currently available:
ATLAS 3.10.2 -> Atlas RPM 3.8.4 is loaded. (-devel on head node). Advise if you wish a module or maintain yourself. (See below for a comment on the dirs in /cbio/ski/fuchs) Boost 1.60.0 -> Boost 1.41 RPMs are loaded. glib -> RPM glib2 2.22 glog -> Confirm please you mean https://code.google.com/archive/p/google-glog/ No RPM loaded can look gflags -> Confirm http://gflags.github.io/gflags/ No RPM loaded can look
Also checking /cbio/ski/fuchs
I do not show the usual dirs were made with your group ownership which I can correct.
The convention is to make a share
, projects
and nobackup
but those were not made. I can do so if you wish to follow that convention or just provide me a name of the dirs you would like.
Oh and leveldb same comment. No RPM. Assume https://github.com/google/leveldb
I made the usual dirs for now. They are group owned and you can do whatever you desire there. If you wish me to build any of the above as modules, just advise.
Oh, and if you build and find a need that an RPM would make easier, I can simply add to puppet if in our repo list.
Just checking here. Are you good with your directories or do you wish items built?
Thanks very much! The goal is to have a shared install of Caffe, which depends on Boost > 1.55 etc http://caffe.berkeleyvision.org/installation.html
Will follow up wrt dirs.
On Thu, Jan 28, 2016 at 10:29 AM, tatarsky notifications@github.com wrote:
Just checking here. Are you good with your directories or do you wish items built?
— Reply to this email directly or view it on GitHub https://github.com/cBio/cbio-cluster/issues/368#issuecomment-176236627.
This would be generally useful if it's OK to make your installation publicly available!
Happy to make a module pointing to a group maintained item if it helps.
I believe from the above statement from @aday00 that they do intend to maintain this software themselves however which ends I believe need for my time to build items. So I can leave this here or close it until you need a system module entry for your tree.
Sounds like a Caffe install would help multiple groups, great! I'm happy to not maintain the software myself/ourselves, and believe modules can disentangle the various Boost versions etc so everyone can use this without any pipelines breaking.
On SuSE systems I used to manage, all this was installed to
/opt/modules/
On Mon, Feb 1, 2016 at 9:19 AM, tatarsky notifications@github.com wrote:
Happy to make a module pointing to a group maintained item if it helps.
I believe from the above statement from @aday00 https://github.com/aday00 that they do intend to maintain this software themselves however which ends I believe need for my time to build items. So I can leave this here or close it until you need a system module entry for your tree.
— Reply to this email directly or view it on GitHub https://github.com/cBio/cbio-cluster/issues/368#issuecomment-177990679.
So to be clear, are you asking me to maintain this software?
Because then I just need an ack from @juanperin he is ok with my time being spent on that or one of his people. I am an hourly contractor.
This is my first time emailing this list. I don't know how billable hours work, I'm not clear to ask for work that costs money, and in the end if we just had a shared directory, that's probably fine, but will follow up soon wrt dirs. I thought asking for software installs would not cost money. If you tell me what it costs etc, I can forward that to my PI. Thanks! -A
On Mon, Feb 1, 2016 at 1:36 PM, tatarsky notifications@github.com wrote:
So to be clear, are you asking me to maintain this software?
Because then I just need an ack from @juanperin https://github.com/juanperin he is ok with my time being spent on that or one of his people. I am an hourly contractor.
— Reply to this email directly or view it on GitHub https://github.com/cBio/cbio-cluster/issues/368#issuecomment-178115459.
Well I speak for my status as a contractor for MSKCC. If you are recharged or not is something I would ask @juanperin . I just note that if I'm doing the work there is indeed a bill and so I want to make sure before I start such work that he is aware of it and can choose other resources if desired. (there are other sysadmins on his team, I am a legacy concept from the SDSC days)
If it was a simple build its usually fine, but this looks a bit elaborate. I'll just wait to hear!
We are adopting the same charging principles applied to all of our HPC resources that charge monthly maintenance fee’s and storage use fee’s. Maintenance fee’s will include some basic level of support in getting help with permissions, small applications, dependencies, etc… But when installs or dependency issues become more time consuming and unique, we will provide an estimate of the time (when greater then 2 hours) in the form of a quote for larger software installations, etc… When applications do in fact affect more then half of the users on the cluster, these costs will be waived in the interest of the community. When the tools are really satisfying only one lab or group, we ask that they try their best to do this in a shared directory.
If you find yourself having much trouble with Caffe, or other applications and want the HPC group to handle the maintenance, just let us know and we’ll give you guys a proper estimate. Otherwise for now it sounds like you have most of what you need to progress, so we’ll wait to hear back.
Thanks! -Juan
On Feb 1, 2016, at 1:36 PM, tatarsky notifications@github.com wrote:
So to be clear, are you asking me to maintain this software?
Because then I just need an ack from @juanperin https://github.com/juanperin he is ok with my time being spent on that or one of his people. I am an hourly contractor.
— Reply to this email directly or view it on GitHub https://github.com/cBio/cbio-cluster/issues/368#issuecomment-178115459.
Normal /cbio/ski/fuchs/{nobackup,projects,share} dirs work well, thanks! I think Thomas will meet Christina Leslie and John Chodera on Feb 2 or 3, to discuss the general utility of a shared Caffe install.
On Mon, Feb 1, 2016 at 1:48 PM, juanperin notifications@github.com wrote:
We are adopting the same charging principles applied to all of our HPC resources that charge monthly maintenance fee’s and storage use fee’s. Maintenance fee’s will include some basic level of support in getting help with permissions, small applications, dependencies, etc… But when installs or dependency issues become more time consuming and unique, we will provide an estimate of the time (when greater then 2 hours) in the form of a quote for larger software installations, etc… When applications do in fact affect more then half of the users on the cluster, these costs will be waived in the interest of the community. When the tools are really satisfying only one lab or group, we ask that they try their best to do this in a shared directory.
If you find yourself having much trouble with Caffe, or other applications and want the HPC group to handle the maintenance, just let us know and we’ll give you guys a proper estimate. Otherwise for now it sounds like you have most of what you need to progress, so we’ll wait to hear back.
Thanks! -Juan
On Feb 1, 2016, at 1:36 PM, tatarsky notifications@github.com wrote:
So to be clear, are you asking me to maintain this software?
Because then I just need an ack from @juanperin < https://github.com/juanperin> he is ok with my time being spent on that or one of his people. I am an hourly contractor.
— Reply to this email directly or view it on GitHub < https://github.com/cBio/cbio-cluster/issues/368#issuecomment-178115459>.
— Reply to this email directly or view it on GitHub https://github.com/cBio/cbio-cluster/issues/368#issuecomment-178124273.
Sounds like a plan! Just let me know. Leaving open!
I heard Christina, John, and Thomas support a shared Caffe install. Is this correct? Does it justify installation without charge? Thanks again! -Andrew
On Mon, Feb 1, 2016 at 5:39 PM, tatarsky notifications@github.com wrote:
Sounds like a plan! Just let me know. Leaving open!
— Reply to this email directly or view it on GitHub https://github.com/cBio/cbio-cluster/issues/368#issuecomment-178235487.
OK. I'm getting the nod to build this. I'd like to use a new set of software areas however based on a GPFS area instead of the legacy "/opt" replications on the local drives. I will still of course make modules to handle the details and sub-builds.
Do you have any objection to my just choosing a location for the items to live in that situation?
Also, Caffe has a fair chunk of Python and they recommend Anaconda.
The convention on this cluster has been users maintain their own personal python and do various includes or builds themselves to add to it dependencies.
Are you wanting a tree specific Python build for this software or do users intend to merge their own trees as needed? If you don't know specifically what I refer to perhaps a call at some point with the people that intend to use this.
To be very clear I intend to follow this suggestion for Python and assume the user has a reasonable personal version unless told otherwise:
To import the caffe Python module after completing the installation, add the module directory to your
$PYTHONPATH by export PYTHONPATH=/path/to/caffe/python:$PYTHONPATH or the like.
You should not import the module in the caffe/python/caffe directory!
No objections. Status here occasionally would be nice if it's not a hassle. Thanks a lot! It's tricky to get this install correct. For instance, as you know, ATLAS tunes itself during installation and has some instructions for disabling CPU throttling. There's also cuDNN to pull from nVidia via free membership, or if you're root you could grab it from (do not post paths). My bin dir has all the dependencies for Caffe, plus other stuff.
I can't make policy changes, and I'm not a Python expert, but Boost.Python is required for Caffe, I believe. I'm not sure how personal pythons interface with Boost.Python, but presumably one could pull in a module so a personal python could use the Boost.Python bindings. So personal pythons as usual I imagine would be fine.
There's also a Matlab interface to Caffe, should just be a configuration option to enable Matlab support in Caffe, and I know for sure there's a Matlab person in lab here. Not sure about other labs. Thanks again very much!
On Wed, Feb 3, 2016 at 4:20 PM, tatarsky notifications@github.com wrote:
To be very clear I intend to follow this suggestion for Python and assume the user has a reasonable personal version unless told otherwise:
To import the caffe Python module after completing the installation, add the module directory to your $PYTHONPATH by export PYTHONPATH=/path/to/caffe/python:$PYTHONPATH or the like. You should not import the module in the caffe/python/caffe directory!
— Reply to this email directly or view it on GitHub https://github.com/cBio/cbio-cluster/issues/368#issuecomment-179474122.
I am extremely busy preparing for node updates. I will work on this but it will be awhile. Just setting expectations.
What branch of the Git source do you want. I show you built rc2.
I'm going to drop some notes as I chew through the prereqs while I do other things.
Boost: Matching your chosen version 1.60.0. Building.
ATLAS: We don't throttle on the Hal cluster. Its disabled and the governor method isn't even in the kernel. Building to a prefix area.
Protobuf attempting from RPM at EPEL. 2.3.0. May still use source as I note you did.
Bleah. Giving up on RPM. Making protobuf matching version module.
2.6.1 for now.
module add protobuf
I'm going to take you up on that cuDNN offer. Can that be shared as a module @aday00 ? I'm going to assume it can but if thats somehow tied to a distribution agreement or something can you let me know? Chunking along in general.
I guess another question as cuDNN looks like they intend for it to layer into /usr/local/cuda7 would be is that desired....for now I will keep it separate. Just trying to reduce the total LD_LIBRARY_PATH needs
going with caffe-rc3
Calling it a day but the main summary of Items I could use an answer to are:
While not all modules and items are done you should be able to see some of the items building up in /cbio/shared/software/
Modules are NOT done for some items. I've manually added items to the Caffe Makefile.
I'm hitting at the moment an NVCC float128 undefined error which I know I've seen before but its been a long day so will try some more tomorrow.
Note to self as I've hit this. It appears the default gcc version causes this issue. https://github.com/BVLC/caffe/issues/1398
Used older gcc for now and will decide if want to build a newer gcc module after some tests.
Also using the rather old Anaconda module as it has the needed python and numpy in a mode this Makefile prefers.
I have a binary that runs but the questions I had need answers to decide if further items are needed in the build. I'll just cut/paste them here from above. I don't know if you use the Git web interface or are reading this via email:
I have made a first draft module: module load caffe
and I am running the test suite as a qlogin job. I am getting many "OK" outputs but I have no experience whatsoever with this code beyond that.
I can see the answer to one of my questions in that many of the example binaries seem to be opencv based so I am working on a rebuild with that integrated but could use a version statement on the opencv desired as it will be a module as well.
Some items that were easier via RPM and version consistent are being runtime added to the nodes. I will verify this draft caffe
binary executes minimally shortly before I rebuild with opencv support.
From my runtest output on a node:
[----------] Global test environment tear-down
[==========] 1792 tests from 252 test cases ran. (482249 ms total)
[ PASSED ] 1792 tests.
So I'm now waiting on my questions before I proceed. Note clearly the current caffe
binary does not have opencv support.
I have rebuilt with an OpenCV 3.0.0 dependency. Please test. Remember you will likely wish to test via an interactive qsub to the gpu queue. Be sure to read and follow the handbook (link on the front of this Git) on proper GPU scheduling etiquette.
$ module load caffe
$ caffe
caffe: command line brew
usage: caffe <command> <args>
commands:
train train or finetune a model
test score a model
device_query show GPU diagnostic information
time benchmark model execution time
There are other binaries. I have no idea how to use the tool ;)
And an item I'm going to state for my reference should we have issues:
the Anaconda module was used for this build. I have a feeling from ldd it has a few older versions of items in its trees including opencv 2.X. While I used opencv 3.0.0 for the build I still see that in the ldd output.
So if there are issues found in that region, I have a feeling a newer Anaconda module is needed. As I have no idea who uses the Anaconda module I would do a fresh one and then rebuild this.
So advise when possible and I'll look at that is needed still.
Hold on a bit on this. I have discovered that the Anaconda libraries also drag in TCL which then breaks the module command ;) So I'm trying another way to build.
OK. I believe back to usable without breaking module itself and also using opencv 3.0.0 and the HDF5 module directly so I'm more confident its closer to what is latest greatest.
Re-running test suite to confirm at least it feels its healthy itself ;)
Thanks very much! Fuchs lab is now testing this Caffe, will follow up.
Cool. I stress I've made a best effort to use the proper settings and rebuilt items I felt it wanted newer than what was already out there. I can review any of those decisions with you but I suspect would make sense verbally.
Let me know!
My pipelines prefer opencv 2.4.9, but opencv 3.0 is the way forward. Hopefully my 2.4.9 stuff doesn't break, haven't tested.
In the past, whenever there's an opencv version conflict, I put the 2.4.9 binaries in the current working directory and it's fine.
Some people prefer opencv 3.0, so I'd say stick with that until it's ever an issue. Thanks again! Super efficient.
I'm happy if it does to make a opencv 2.4.9 module and recompile. I just noted that the system RPM version was 2.0 which was not supported. I will be making a formal module for the opencv/3.0.0 that I did. Its currently located at: /cbio/shared/software/opencv/3.0.0
but I'll add a module for it for separate use.
You are welcome. Was an interesting build.
Really, you're tremendous. And opencv 3.0, even 3.1.0, is the way forward. If your interest persists, opencv isn't actually complete. There's a contrib component too, and the contrib build requires a rebuild of opencv, pretty tight integration: https://github.com/itseez/opencv_contrib
Yeah, I actually gleaned large parts of my module layout from that doc. The main issue is the SDSC built modules for some items are getting pretty old. And the Anaconda one was linked to older OpenCV and some older HDF5. So I'm probably going to rev a few more items in the end.
I'm also fixing the default gcc we use as I did note a bug in the compile was fixed with 4.8.5. More on this will appear as open Git issues here in this repo so turn on that watch flag for it.
Appreciate the thanks.
If you choose to install opencv_contrib also, which would help me (and perhaps others), i'll note the opencv and opencv_contrib release version should match. So if opencv is 3.0.0, opencv_contrib should also be 3.0.0. The opencv_contrib 3.1.0 expects some headers etc not present in opencv 3.0.0, for example.
OK. I'll take a look at that. Do your tests and when its looking at least mostly on the right track I'll rev the opencv as above stated and rebuild caffe with it.
Hi HPC Request,
Juan tells me to ask here for a Fuchs-Lab-wide group directory for software installs on the HAL cluster.
We have several libraries to install, including ATLAS 3.10.2, Boost 1.60.0, Boost.Python, glib, glog, gflags, leveldb, and others.
It would be really nice to install as many libraries as possible from RPMs (with modules to load/unload accordingly, which may be of HAL-cluster-wide interest), but if none of that's supported, we'll roll our own installs from source.
Thanks in advance! schaumba@mskcc