search

GeoscienceAustralia / dea-notebooks

Repository for Digital Earth Australia Jupyter Notebooks: tools and workflows for geospatial analysis with Open Data Cube and Xarray
https://docs.dea.ga.gov.au/notebooks/
Apache License 2.0
448 stars 128 forks source link

Update doco to reflect decommissioning of strudel/VDI #845

Open CEKrause opened 3 years ago

CEKrause commented 3 years ago

Describe the bug/issue Need to update the README doco and wiki with new instructions for accessing NCI OOD service (no longer via strudel).

CEKrause commented 3 years ago

Some existing instructions that might be helpful? https://opus.nci.org.au/display/OOD/NCI%27s+dk92+Data+Analysis+Notebooks

jwurtzel commented 3 years ago

Just some comments from my initial trials with @zhuq56 In order to use Jupyter Lab instance with dea modules, the module needs to be preloaded as part of the setup. image

This loads to the home directory. We are still not sure how to launch from the preferred working directory (ie. g/data/r78/user/dea-notebooks). We tried to 'cd' to the directory in the Jupyter notebook itself, but relative paths (e.g. appending the /Scripts/ folder) still weren't working. We have tried hard-coding the paths instead and this seems to work.

We are also still trying to find the right size machine for our job. The VDI was 8 CPUs, I think? We note that there is now a waiting time to launch an instance (though smaller machines load pretty quickly). We will report back in later when we have successfully completed our job.

robbibt commented 3 years ago

Yeah, this would be great to have nicely documented.

@jwurtzel: out of interest, why use the OOD JupyterLab launcher when you could use the OOD VDI launcher which should work the same way it used to? (just curious, I have no reason to think either option is better than the other, the VDI one just seemed more familiar when I logged in for the first time the other day)

jwurtzel commented 3 years ago

@robbibt : To be honest, @zhuq56 is taking the lead on this, but I think she went that way because that's what the NCI documentation examples showed. I haven't tried the VDI launcher. Not sure if she has. No clue if there is any difference in performance, but maybe using that will solve some of the path issues. It looks like the way dask works in Jupyter Lab is different. If that hasn't changed on the VDI, we may be better off there. We'll get back to you once we've tried both.

robbibt commented 3 years ago

Yep, OK! At the moment, it looks like the VDI launcher is a close copy of the original VDI which will hopefully make transitioning easy - it does however look like the JupyterLab launcher supports some extra things like possibly being able to launch a dask cluster from Gadi itself which would be cool - definitely interested in any comparisons you make between them!

zhuq56 commented 3 years ago

Morning Robbi,

We did the processing in OOD Jupyter launcher and it works!

Another question is tracking the long processing step via the Dask cluster. Tried to make the add the dask extension following https://opus.nci.org.au/display/OOD/3.2.1.+JupyterLab+Basics

but back with the error “communicating with server extension. Consult the documentation for how to ensure that it is enabled.” TypeError: Failed to fetch

Wondering if any other options Dask from OOD, or let me know if missed any steps.

Cheers Esther

From: Robbi Bishop-Taylor @.> Sent: Monday, 2 August 2021 12:14 PM To: GeoscienceAustralia/dea-notebooks @.> Cc: Esther Zhu @.>; Mention @.> Subject: Re: [GeoscienceAustralia/dea-notebooks] Update doco to reflect decommissioning of strudel/VDI (#845)

Yep, OK! At the moment, it looks like the VDI launcher is a close copy of the original VDI which will hopefully make transitioning easy - it does however look like the JupyterLab launcher supports some extra things like possibly being able to launch a dask cluster from Gadi itself which would be cool - definitely interested in any comparisons you make between them!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/GeoscienceAustralia/dea-notebooks/issues/845#issuecomment-890660208, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANUDXZRDHCQXHLUXSNXPRKTT2X5OVANCNFSM5BLXSFSA.