djbard
commented
7 years ago We already have Jupyter supported at NERSC - and we use it extensively! See: http://www.nersc.gov/users/data-analytics/data-analytics/web-applications-for-data-analytics/ https://ipython.nersc.gov/hub/login It's basically our new favourite toy. Service may be a little spotty right now, as the server sits on one of the Cori nodes and that system is down for the next few weeks, but we can certainly try things out once the system integration is complete. We even have a (frankly amazing) interface with our batch system - so you can submit batch jobs directly from the notebook. We had a number of live demos of this at our Data Day last month that people might be interested in. (If you're really interested in how we have things running, you can see out github and Dockerhub repos).
drphilmarshall
heather999
@richardxdubois @dodelson @connolly @djbard
@jchiang87 @tony-johnson and I met with the IPAC DM SUIT (science user interface and tools) team today (Xiuqin, Gregory, Trey, Tatiana). They were visiting the Camera team, and I'd said that it would be interesting to talk more with them after I chatted a bit to them in Tucson.
Jim and I talked them through what we think is a fairly typical use case, https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/297 In that issue, the Twinkles task force collaboratively analyzed a mixture of Twinkles DM level 2 catalog tables, OpSim and PhoSim tables, and DM Level 2 image cutouts to diagnose a simple problem (in this case, omission of CR removal from the processing pipeline caused occasional loss of psfFlux). With the SUIT team, we made a whiteboard list of all the things we did in issue 297, and concluded that we could have carried out this whole analysis more efficiently if we could have accessed the Twinkles data through a system like the SUIT. Here's the whiteboard:
They explained that they won't have any remote-accessible jupyter notebooks running at their prototype DAC at NCSA until next spring, when they do have that running, it seems as though we might be able to set up a bespoke SUIT system at NERSC to support DESC access to DESC data. It's not clear how much support the project would be able to provide, but it was good to start the conversation.
Meanwhile, it seems clear that one key component of making a SUIT work at NERSC is to have a JupyterHub running there, enabling notebook access to NERSC data. This would already be really useful for us! All the SUIT would then do is provide extra widgets to run in those notebooks (like the firefly image viewer, and so on). But we could do a great deal with just numpy, scipy, astropy etc running in notebooks at NERSC but coded remotely by DESC members. Is a jupyter hub something we should therefore work to get set up at NERSC? Do you know of any NERSC users already doing this, Debbie?