search

jcbird / ppv

Tools for dealing with SDSS-V plate files and plate runs.
BSD 3-Clause "New" or "Revised" License
1 stars 0 forks source link

error with intial import #6

Closed jcbird closed 4 years ago

jcbird commented 4 years ago

I guess I should post an error log - after entering the password, and then interrupting, this is what it spits out:

An error occurred. /Users/mkounkel/research/cluster/targeting/ppv/plate/platePlans_sdss5.fits does not exist

AND an error occured when trying to execute either

util.download.plate_plans OR _parse_plate_plans

FileNotFoundError Traceback (most recent call last)

in 1 from astropy.table import Table ----> 2 import ppv # the FIRST time you import ppv, the platePlans.par file is downloaded 3 # you can ignore the WARNING ~/opt/anaconda3/lib/python3.7/importlib/_bootstrap.py in _find_and_load(name, import_) ~/opt/anaconda3/lib/python3.7/importlib/_bootstrap.py in _find_and_load_unlocked(name, import_) ~/opt/anaconda3/lib/python3.7/importlib/_bootstrap.py in _load_unlocked(spec) ~/opt/anaconda3/lib/python3.7/importlib/_bootstrap.py in _load_backward_compatible(spec) ~/opt/anaconda3/lib/python3.7/site-packages/ppv-0.0.1-py3.7.egg/ppv/__init__.py in 124 125 --> 126 ppv = Summary() ~/opt/anaconda3/lib/python3.7/site-packages/ppv-0.0.1-py3.7.egg/ppv/__init__.py in __init__(self) 13 Constructor 14 """ ---> 15 self._load() 16 17 def __str__(self): ~/opt/anaconda3/lib/python3.7/site-packages/ppv-0.0.1-py3.7.egg/ppv/__init__.py in _load(self) 30 Load summary and parse it 31 """ ---> 32 allplate = data_io.load_plansummary() 33 allplate.add_index('name') # for quick filtering on fieldname 34 self._allplate = allplate ~/opt/anaconda3/lib/python3.7/site-packages/ppv-0.0.1-py3.7.egg/ppv/data/io.py in load_plansummary() 59 else: 60 _parse_plate_plans() # already there, parse it ---> 61 return Table.read(os.fspath(paths.plate_plans()), format='fits') 62 63 ~/opt/anaconda3/lib/python3.7/site-packages/astropy/table/connect.py in __call__(self, *args, **kwargs) 50 def __call__(self, *args, **kwargs): 51 cls = self._cls ---> 52 out = registry.read(cls, *args, **kwargs) 53 54 # For some readers (e.g., ascii.ecsv), the returned `out` class is not ~/opt/anaconda3/lib/python3.7/site-packages/astropy/io/registry.py in read(cls, format, *args, **kwargs) 521 522 reader = get_reader(format, cls) --> 523 data = reader(*args, **kwargs) 524 525 if not isinstance(data, cls): ~/opt/anaconda3/lib/python3.7/site-packages/astropy/io/fits/connect.py in read_table_fits(input, hdu, astropy_native, memmap, character_as_bytes) 210 211 hdulist = fits_open(input, character_as_bytes=character_as_bytes, --> 212 memmap=memmap) 213 214 try: ~/opt/anaconda3/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py in fitsopen(name, mode, memmap, save_backup, cache, lazy_load_hdus, **kwargs) 163 164 return HDUList.fromfile(name, mode, memmap, save_backup, cache, --> 165 lazy_load_hdus, **kwargs) 166 167 ~/opt/anaconda3/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py in fromfile(cls, fileobj, mode, memmap, save_backup, cache, lazy_load_hdus, **kwargs) 403 return cls._readfrom(fileobj=fileobj, mode=mode, memmap=memmap, 404 save_backup=save_backup, cache=cache, --> 405 lazy_load_hdus=lazy_load_hdus, **kwargs) 406 407 @classmethod ~/opt/anaconda3/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py in _readfrom(cls, fileobj, data, mode, memmap, save_backup, cache, lazy_load_hdus, **kwargs) 1052 if not isinstance(fileobj, _File): 1053 # instantiate a FITS file object (ffo) -> 1054 fileobj = _File(fileobj, mode=mode, memmap=memmap, cache=cache) 1055 # The Astropy mode is determined by the _File initializer if the 1056 # supplied mode was None ~/opt/anaconda3/lib/python3.7/site-packages/astropy/utils/decorators.py in wrapper(*args, **kwargs) 533 warnings.warn(message, warning_type, stacklevel=2) 534 --> 535 return function(*args, **kwargs) 536 537 return wrapper ~/opt/anaconda3/lib/python3.7/site-packages/astropy/io/fits/file.py in __init__(self, fileobj, mode, memmap, overwrite, cache) 191 self._open_fileobj(fileobj, mode, overwrite) 192 elif isinstance(fileobj, str): --> 193 self._open_filename(fileobj, mode, overwrite) 194 else: 195 self._open_filelike(fileobj, mode, overwrite) ~/opt/anaconda3/lib/python3.7/site-packages/astropy/io/fits/file.py in _open_filename(self, filename, mode, overwrite) 572 573 if not self._try_read_compressed(self.name, magic, mode, ext=ext): --> 574 self._file = fileobj_open(self.name, IO_FITS_MODES[mode]) 575 self.close_on_error = True 576 ~/opt/anaconda3/lib/python3.7/site-packages/astropy/io/fits/util.py in fileobj_open(filename, mode) 394 """ 395 --> 396 return open(filename, mode, buffering=0) 397 398 FileNotFoundError: [Errno 2] No such file or directory: '/Users/mkounkel/research/cluster/targeting/ppv/plate/platePlans_sdss5.fits' _Originally posted by @mkounkel in https://github.com/jcbird/ppv/issues/1#issuecomment-704031387_
jcbird commented 4 years ago

I am moving this to a new issue @mkounkel .

Could you try to pull the latest commit and reinstall via python setup install ? Then report back. Thanks!

jcbird commented 4 years ago

Also, restart the kernel in the Jupyter notebook to clear everything from memory.

mkounkel commented 4 years ago

No change from yesterday, even after a complete uninstall to test that it wasn't doing some weird caching. Installing it back again still leaves it hanging after trying to import.

jcbird commented 4 years ago

Sorry for the delay @mkounkel . OK, I just did a fresh pull and a brand new install in a new environment. No issues. Could you paste/attach the contents of your ppv_setup.ini file?

Also, please try deleting the ppv directory in site-packages (uninstalling ppv). Do a new pull of the ppv repo, reinstall, and report back. If you get the same error, could you try reading in any other fits file with astropy and make sure the problem isn't there. Thanks! I would be more inclined to think ppv is at fault but your experience is the only one I have heard like this so far, so ensuring astropy is working properly is worth a shot, too.

mkounkel commented 4 years ago

One thing I do not do is set up a separate environment for all the projects - I have no idea how to manage them. I'm also still running 3.7, not 3.8

As before, I uninstalled the package (via pip uninstall ppv, can confirm that afterwards it can't find the package). Installing it back again - same issue as before. The password I'm entering is correct, copypasting the same string from the password manager works fine in ssh-ing directly. I tried entering intentionally incorrect password when logging in, it didn't call me out on it, just still hanged up in exact same way. I tried changing the password, thinking that perhaps some of the special symbols might be messing it up - but, no, same issue as before.

And, no, pretty sure that the issue is just with logging in to download the files to begin with, not with astropy.

I was able to parse the files in the five_plates repo directly, I think, so it's not a be-all end-all

Contents of the ini file:

[paths]
plate_dir = /Users/mkounkel/research/cluster/targeting/ppv/plate
fiveplates_dir = /Users/mkounkel/research/cluster/targeting/ppv/plate1

# If you want your plate files in, e.g., /home/user/data/plates/0150XX
# and /home/user/data/plates/0160XX, then edit the plate_dir line to read
# plate_dir = /home/user/data/plates

# fiveplates_dir is the directory corresponsing to 'plateruns' in
# your local five_plates repo (you need to clone this repo, see ppv README)
# e.g., if the five_plates repo is located in /home/user/local, then put
# fiveplates_dir = /home/user/local/five_plates/plateruns

[username]
sdss_org = u6031723
# NO quotes
# If you do NOT have an account at Utah, please get one at
# https://wiki.sdss.org/display/DATA/Utah+Accounts
jcbird commented 4 years ago

Did you use pip to install ppv or did you use python setup install (as you should). If you used python setup install, I am 99% convinced that using pip uninstall to uninstall a package that pip didn't install is the issue. I think remove the .egg file and directory from site-packages will do the trick. If that isn't clear, I will get some step by step instructions to you when time permits.

mkounkel commented 4 years ago

I used python setup install. Using pip to uninstall something I installed via other methods never was an issue in any other circumstances before - it just finds and deletes the egg file from site-packages.

alexji commented 4 years ago

First of all, can I be added to fiveplates?

I am having the same issue as Marina. Here's what I did:

It asks me for my Utah password and then hangs.

Enter password for u6031721:
Password: <enter password and push enter and nothing happens>

I played around to see where it hangs, and it is at the wait. I suspect somehow an endline isn't being sent, even though that's explicitly what sendline is supposed to do...

def _run_rsync(rsync_command_string, utah_passwd):
    ## shell_path = os.environ.get('SHELL')
    ## rsync_child = pexpect.spawn(shell_path, ['-c', rsync_command_string])
    rsync_child = pexpect.spawn(rsync_command_string)
    rsync_child.expect('[P/p]assword:')
    rsync_child.sendline(utah_passwd)
    rsync_child.wait() # <-- it gets to here and stops, regardless of what I put in for a password
    rsync_child.close()
    ...

My version of pexpect is 4.8.0. I am on a Mac with iTerm2 but also checked with the Mac default Terminal program that also fails. I do have an older version of conda, though that shouldn't matter.

Note that I can login myself, and also if I directly call the rsync command that it prints out initially, that does work, so it's something about the rsync command.

Thanks!

jcbird commented 4 years ago

Oh goodness! Thank you Alex! I was so focus on the error message from Marina I glossed over that she had to interrupt the password! That narrows things down considerably. Marina, can you check the version of pexpect you are using? Thanks ! I should be able to fix this today.

On Sun, Oct 11, 2020, 5:47 PM Alex Ji notifications@github.com wrote:

First of all, can I be added to fiveplates?

I am having the same issue as Marina. Here's what I did:

  • grab the latest version of the master ppv branch
  • create a new environment using ppv_sdss_min.yml
  • Update the plate_dir, fiveplates_dir, and sdss_org in ppv_setup.ini and copy that to ~/.config
  • Run python setup.py install
  • Open ipython and import ppv

It asks me for my Utah password and then hangs.

Enter password for u6031721: Password:

I played around to see where it hangs, and it is at the wait. I suspect somehow an endline isn't being sent, even though that's explicitly what sendline is supposed to do...

def _run_rsync(rsync_command_string, utah_passwd):

shell_path = os.environ.get('SHELL')

## rsync_child = pexpect.spawn(shell_path, ['-c', rsync_command_string])
rsync_child = pexpect.spawn(rsync_command_string)
rsync_child.expect('[P/p]assword:')
rsync_child.sendline(utah_passwd)
rsync_child.wait() # <-- it gets to here and stops, regardless of what I put in for a password
rsync_child.close()
...

My version of pexpect is 4.8.0. I am on a Mac with iTerm2 but also checked with the Mac default Terminal program that also fails. I do have an older version of conda, though that shouldn't matter.

Note that I can login myself, and also if I directly call the rsync command that it prints out initially, that does work, so it's something about the rsync command.

Thanks!

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jcbird/ppv/issues/6#issuecomment-706772719, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRADE7J3CK6XHVQB27HLNLSKIRWVANCNFSM4SGHI5CA .

mkounkel commented 4 years ago

I had a version 4.7, just upgraded it to 4.8 without improvement.

jcbird commented 4 years ago

Thanks! I'll have a fix soon once I have a chance in the next few hours.

On Mon, Oct 12, 2020, 12:08 PM mkounkel notifications@github.com wrote:

I had a version 4.7, just upgraded it to 4.8 without improvement.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jcbird/ppv/issues/6#issuecomment-707210790, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRADE3LO2KXR7SCBUUFI53SKMSWTANCNFSM4SGHI5CA .

jcbird commented 4 years ago

Narrowing it down, Alex, Marina, what version of Mac OS do you use?

On Mon, Oct 12, 2020, 12:11 PM J. Bird jonathan.bird.io@gmail.com wrote:

Thanks! I'll have a fix soon once I have a chance in the next few hours.

On Mon, Oct 12, 2020, 12:08 PM mkounkel notifications@github.com wrote:

I had a version 4.7, just upgraded it to 4.8 without improvement.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jcbird/ppv/issues/6#issuecomment-707210790, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRADE3LO2KXR7SCBUUFI53SKMSWTANCNFSM4SGHI5CA .

mkounkel commented 4 years ago

10.15.5

alexji commented 4 years ago

I have 10.12.6 (yes I know I'm a dinosaur)

jcbird commented 4 years ago

OK, hopefully some progress. Can @alexji and @mkounkel do the following and report back?

git pull # get the latest commits git checkout rsync_fix # switch to the testing branch for this fix python setup install # install the latest package import ppv # import and do the first test

and follow up with whatever you want. I do think it's the pexpect.wait that is acting differently on different platforms. That has been changed.

Thanks and let me know!

mkounkel commented 4 years ago

Works now!

jcbird commented 4 years ago

Woohoo! I'll merge this to master in a sec so you won't need to switch branches.

On Mon, Oct 12, 2020, 3:33 PM mkounkel notifications@github.com wrote:

Works now!

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/jcbird/ppv/issues/6#issuecomment-707306536, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRADEZYW563HYHNQJW3KS3SKNKXTANCNFSM4SGHI5CA .

alexji commented 4 years ago

Great it works on master for me! @jcbird I also mentioned before about not having access to five_plates, is there someone I should contact?

jcbird commented 4 years ago

Awesome! It's merged into master as @alexji saw. I am going to close this one. Thanks for your help, everyone!

@alexji : for now, I will ping Joel on getting access to five_plates for you. Luckily, you do NOT need five_plates access for the 2020.10.a.mwm-bhm platerun.

Rock on!