Jupyter notebook & password

[mkounkel]

There is one bug of minor annoyance for rsync - if I run it from the terminal, it's fine, because it waits for Utah password. Jupyter notebook, on the other hand, skips entering the password, so it crashes when trying to run maybe_observed = planet_targets.available_in(plate_run_2020_08_c)

[deleenm]

I have the same problem!

[jcbird]

Thanks @mkounkel and @deleenm ! I am hoping to address this later today.

Could you tell me what version of jupyter you are using and if you are using jupyter notebook or jupyter-lab? Thanks!

[deleenm]

I am using Jupyter_Hub server 1.0.0 to serve a standard jupyter notebook. With the following conda environmnet for the ipython kernel:

ipykernel 5.3.4 py38h23f93f0_0 conda-forge ipython 7.18.1 py38h1cdfbd6_0 conda-forge

And this conda environment for the jupyter_hub server itself: jupyter 1.0.0 py37_7
jupyter_client 5.3.4 py37_0
jupyter_console 6.1.0 py_0
jupyter_core 4.6.1 py37_0
jupyterhub 1.0.0 py37_0
jupyterlab 1.2.6 pyhf63ae98_0
jupyterlab_server 1.0.6 py_0

[jcbird]

Thanks! Very helpful!

On Thu, Oct 1, 2020, 4:39 PM Nathan De Lee notifications@github.com wrote:

I am using Jupyter_Hub server 1.0.0 to serve a standard jupyter notebook. With the following conda environmnet for the ipython kernel:

ipykernel 5.3.4 py38h23f93f0_0 conda-forge ipython 7.18.1 py38h1cdfbd6_0 conda-forge

And this conda environment for the jupyter_hub server itself: jupyter 1.0.0 py37_7 jupyter_client 5.3.4 py37_0 jupyter_console 6.1.0 py_0 jupyter_core 4.6.1 py37_0 jupyterhub 1.0.0 py37_0 jupyterlab 1.2.6 pyhf63ae98_0 jupyterlab_server 1.0.6 py_0

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jcbird/ppv/issues/1#issuecomment-702412369, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRADE7VFDIVI3QZX74YUPTSITZI5ANCNFSM4R2QBA2Q .

[mkounkel]

Similar on my end, running jupyter notebook, not lab. Python 3.7, jupyter 1.0, jupyter-core 4.5.0, jupyter-client 5.3.3, jupyter-console 6.0.0, ipython 7.8

[deleenm]

This may be a useful way to get passwords in jupyter notebooks: import getpass password = getpass.getpass()

[deleenm]

Once you have the password in a variable, you can feed it to rsync etc. Also, you should only need to do this once per kernel session with the notebook, since you can keep the password in memory.

[jcbird]

Thanks all. @deleenm appreciate the suggestion. Unfortunately, rsync won't accept passwords on the command line. It can use ssh to do the authentication, but Utah won't allow identity keys (BOOOOO). I will get a separate script up today that you can run in a terminal just once before an analysis session. The script will update the plateplans.par file and ALL available plugholes files (in case those were revised).

Going forward, I am going to incorporate https://github.com/sdss/sdss_access from SDSS . That way, you will only need a .netrc file and will never have to input your password again.

[deleenm]

Another option would be to use Popen and feed the password into rsync via sshpass. Here is a website with a possible version of that: https://stackoverflow.com/questions/21379185/python-subprocess-rsync-using-sshpass-and-specify-port

[deleenm]

@jcbird Here is maybe an even easier option using an environmental variable or password file to run rsync: https://stackoverflow.com/questions/13306842/how-to-parse-server-password-with-subprocess-popen-in-django/13309144

[jcbird]

Thanks! I had already been playing with the shell=True option. Your last link looks good. I don't LOVE it, and I think i will try to incorporate sdss_access soon, but this will work for now. Stay tuned!

[deleenm]

That was my feeling as well. I don't like that you basically have to write the password in plain text to either an environment variable or file, but it is good enough for the moment. I can't believe that rsync won't just let you pass it a password on the command line. It feels like one of those security "features" that forces you to do less secure things.

[jcbird]

Thanks for all the feedback @deleenm and @mkounkel !

I am using the pexpect package now. This is the default package that ipython uses for this sort of thing. SO all should be well now.

Check out the new ppv.ppv.update_platefiles() command (see ipynb tutorial and README) . It will grab ALL the latest from Utah in one go too!

Please comment back here after trying the latest commit so I can close the issue. Thanks!

[deleenm]

Looks like it worked for me!

[jcbird]

I'm going to close this one. @mkounkel , let me know if you have any problems.

[mkounkel]

Sorry, I think the update broke it for me completely. Now I can get the password prompt, but it just never does anything after it, either in jupyter or in the terminal. I can run the rsync command it shows outside of python environment, but not within it for whatever reason

[deleenm]

I was able to run both tutorials without issue. The rsync definitely ran for me. I even deleted all of the files in my plates directory, and the notebook repopulated them. Note I did have to enter my password twice once for: import ppv and once for: ppv.ppv.update_platefiles()

[mkounkel]

I've deleted the files, and it's stuck for me on import ppv. It's been trying to run for over 10 minutes now, I assume it shouldn't take this long. When I had older files, that part worked fine without a password prompt, but the moment it did prompt for a password, it got stuck again.

[mkounkel]

I guess I should post an error log - after entering the password, and then interrupting, this is what it spits out:

An error occurred. /Users/mkounkel/research/cluster/targeting/ppv/plate/platePlans_sdss5.fits does not exist

AND an error occured when trying to execute either

util.download.plate_plans OR _parse_plate_plans

---

FileNotFoundError Traceback (most recent call last)

in 1 from astropy.table import Table ----> 2 import ppv # the FIRST time you import ppv, the platePlans.par file is downloaded 3 # you can ignore the WARNING ~/opt/anaconda3/lib/python3.7/importlib/_bootstrap.py in _find_and_load(name, import_) ~/opt/anaconda3/lib/python3.7/importlib/_bootstrap.py in _find_and_load_unlocked(name, import_) ~/opt/anaconda3/lib/python3.7/importlib/_bootstrap.py in _load_unlocked(spec) ~/opt/anaconda3/lib/python3.7/importlib/_bootstrap.py in _load_backward_compatible(spec) ~/opt/anaconda3/lib/python3.7/site-packages/ppv-0.0.1-py3.7.egg/ppv/__init__.py in 124 125 --> 126 ppv = Summary() ~/opt/anaconda3/lib/python3.7/site-packages/ppv-0.0.1-py3.7.egg/ppv/__init__.py in __init__(self) 13 Constructor 14 """ ---> 15 self._load() 16 17 def __str__(self): ~/opt/anaconda3/lib/python3.7/site-packages/ppv-0.0.1-py3.7.egg/ppv/__init__.py in _load(self) 30 Load summary and parse it 31 """ ---> 32 allplate = data_io.load_plansummary() 33 allplate.add_index('name') # for quick filtering on fieldname 34 self._allplate = allplate ~/opt/anaconda3/lib/python3.7/site-packages/ppv-0.0.1-py3.7.egg/ppv/data/io.py in load_plansummary() 59 else: 60 _parse_plate_plans() # already there, parse it ---> 61 return Table.read(os.fspath(paths.plate_plans()), format='fits') 62 63 ~/opt/anaconda3/lib/python3.7/site-packages/astropy/table/connect.py in __call__(self, *args, **kwargs) 50 def __call__(self, *args, **kwargs): 51 cls = self._cls ---> 52 out = registry.read(cls, *args, **kwargs) 53 54 # For some readers (e.g., ascii.ecsv), the returned `out` class is not ~/opt/anaconda3/lib/python3.7/site-packages/astropy/io/registry.py in read(cls, format, *args, **kwargs) 521 522 reader = get_reader(format, cls) --> 523 data = reader(*args, **kwargs) 524 525 if not isinstance(data, cls): ~/opt/anaconda3/lib/python3.7/site-packages/astropy/io/fits/connect.py in read_table_fits(input, hdu, astropy_native, memmap, character_as_bytes) 210 211 hdulist = fits_open(input, character_as_bytes=character_as_bytes, --> 212 memmap=memmap) 213 214 try: ~/opt/anaconda3/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py in fitsopen(name, mode, memmap, save_backup, cache, lazy_load_hdus, **kwargs) 163 164 return HDUList.fromfile(name, mode, memmap, save_backup, cache, --> 165 lazy_load_hdus, **kwargs) 166 167 ~/opt/anaconda3/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py in fromfile(cls, fileobj, mode, memmap, save_backup, cache, lazy_load_hdus, **kwargs) 403 return cls._readfrom(fileobj=fileobj, mode=mode, memmap=memmap, 404 save_backup=save_backup, cache=cache, --> 405 lazy_load_hdus=lazy_load_hdus, **kwargs) 406 407 @classmethod ~/opt/anaconda3/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py in _readfrom(cls, fileobj, data, mode, memmap, save_backup, cache, lazy_load_hdus, **kwargs) 1052 if not isinstance(fileobj, _File): 1053 # instantiate a FITS file object (ffo) -> 1054 fileobj = _File(fileobj, mode=mode, memmap=memmap, cache=cache) 1055 # The Astropy mode is determined by the _File initializer if the 1056 # supplied mode was None ~/opt/anaconda3/lib/python3.7/site-packages/astropy/utils/decorators.py in wrapper(*args, **kwargs) 533 warnings.warn(message, warning_type, stacklevel=2) 534 --> 535 return function(*args, **kwargs) 536 537 return wrapper ~/opt/anaconda3/lib/python3.7/site-packages/astropy/io/fits/file.py in __init__(self, fileobj, mode, memmap, overwrite, cache) 191 self._open_fileobj(fileobj, mode, overwrite) 192 elif isinstance(fileobj, str): --> 193 self._open_filename(fileobj, mode, overwrite) 194 else: 195 self._open_filelike(fileobj, mode, overwrite) ~/opt/anaconda3/lib/python3.7/site-packages/astropy/io/fits/file.py in _open_filename(self, filename, mode, overwrite) 572 573 if not self._try_read_compressed(self.name, magic, mode, ext=ext): --> 574 self._file = fileobj_open(self.name, IO_FITS_MODES[mode]) 575 self.close_on_error = True 576 ~/opt/anaconda3/lib/python3.7/site-packages/astropy/io/fits/util.py in fileobj_open(filename, mode) 394 """ 395 --> 396 return open(filename, mode, buffering=0) 397 398 FileNotFoundError: [Errno 2] No such file or directory: '/Users/mkounkel/research/cluster/targeting/ppv/plate/platePlans_sdss5.fits'