GAIA ID

[yaelnaze]

In the example with a TESS LC, star = eleanor.Source(gaia=4675352109658261376, sector=1) -> is that GAIA DR1 or DR2 identifier ? There are not identical...

[afeinstein20]

Hi @yaelnaze, We're using Gaia DR2 identifiers. I'll add a comment in the example notebook to avoid future confusion :)

[yaelnaze]

Hello, OK, thanks - but is that the only thing to change in the quickstart tutorial, or are there restrictions on python version or whatever ? I've tried here those exact commands, on several computers, and it fails...

1/ data = eleanor.TargetData(star, height=15, width=15, bkg_size=31, do_psf=True, do_pca=True) results in "WARNING: FITSFixedWarning: 'datfix' made the change 'Set MJD-OBS to 58324.812768 from DATE-OBS. Set DATE-BEG to '2018-07-25T19:31:32.353' from MJD-BEG'. [astropy.wcs.wcs] WARNING: The "exact" method is not yet implemented for rectangular apertures -- using "subpixel" method with "subpixels=32" [photutils.aperture.core] ... numpy.linalg.LinAlgError: Singular matrix"

2/ star = eleanor.Source(gaia=4675352109658261376, sector=1) results in " Created TAP+ (v1.2.1) - Connection: Host: gea.esac.esa.int Use HTTPS: True Port: 443 SSL Port: 443 Created TAP+ (v1.2.1) - Connection: Host: geadata.esac.esa.int Use HTTPS: True Port: 443 SSL Port: 443 ... ValueError: 1:0: no element found"

[benmontet]

Hi @yaelnaze, can I ask what versions of eleanor, astroquery, and astropy you are using? You can check these by importing the module and then typing eleanor.__version__ for example, with two underscores on either side.

If they are anything less than 1.0.1, 0.3.10, and 4.0.0 can you please update those modules and try again? The following calls worked fine on my local machine:

s = eleanor.Source(gaia=4675352109658261376, sector=1) data = eleanor.TargetData(s, height=15, width=15, bkg_size=31, do_psf=True, do_pca=True)

[yaelnaze]

Versions are ok - all more than what you mentioned.

We have uninstalled, re-installed, uninstalled, ... and we could make it work sometimes (but not always). It seems like eleanor.Source task sometimes download a lot (**) and sometimes do nothing and sometimes only download the last three files but not the first ones - even if, in all cases, the .eleanor directory was cleaned before doing anything ! The most common thing being the download of the last three files, which gives the impression it's ok but results in singular matrix when you do data = eleanor.TargetData...

cheers,

---

** This is the first light curve you have made for this sector. Getting eleanor metadata products for Sector 1... This will only take a minute, and only needs to be done once. Any other light curves you make in this sector will be faster. % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1991k 100 1991k 0 0 1703k 0 0:00:01 0:00:01 --:--:-- 1703k Target Acquired Cadences Calculated Quality Flags Assured % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 4997k 100 4997k 0 0 3592k 0 0:00:01 0:00:01 --:--:-- 3590k ... CBVs Made Success! Sector 1 now available. Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:HLSP/eleanor/postcards/s0001/4-1/hlsp_eleanor_tess_ffi_postcard-s0001-4-1-cal-0902-1078_tess_v2_bkg.fits to /home/detal/.eleanor/mastDownload/HLSP/hlsp_eleanor_tess_ffi_postcard-s0001-4-1-cal-0902-1078_tess_v2_pc/hlsp_eleanor_tess_ffi_postcard-s0001-4-1-cal-0902-1078_tess_v2_bkg.fits ... |=============================================================| 78M/ 78M (100.00%) 12s Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:HLSP/eleanor/postcards/s0001/4-1/hlsp_eleanor_tess_ffi_postcard-s0001-4-1-cal-0902-1078_tess_v2_pc.fits to /home/detal/.eleanor/mastDownload/HLSP/hlsp_eleanor_tess_ffi_postcard-s0001-4-1-cal-0902-1078_tess_v2_pc/hlsp_eleanor_tess_ffi_postcard-s0001-4-1-cal-0902-1078_tess_v2_pc.fits ... |=============================================================| 158M/158M (100.00%) 14s Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:HLSP/eleanor/postcards/s0001/4-1/hlsp_eleanor_tess_ffi_postcard-s0001-4-1_tess_v2_pm.txt to /home/detal/.eleanor/mastDownload/HLSP/hlsp_eleanor_tess_ffi_postcard-s0001-4-1-cal-0902-1078_tess_v2_pc/hlsp_eleanor_tess_ffi_postcard-s0001-4-1_tess_v2_pm.txt ... |=============================================================| 237k/237k (100.00%) 0s

[afeinstein20]

Yup this is to be expected! In order to move eleanor towards being more automated, we create the quality flags, cadence numbers, and some other metadata directly on your machine. Once you have these files for a given sector-camera-ccd, it skips these steps for other targets on the same combination.

The files being downloaded from MAST are the sector-camera-ccd pointing model (which cannot be made locally, unless you want to download all of the FFIs!), the postcard the target falls on, and the 2D modeled background for that postcard.

Are you getting the same issues as you posted before, when using the same call and after emptying the .eleanor directory?

[benmontet]

I agree with @afeinstein20, the first downloading should happen for the first star you make a LC for in a given sector after a clean install, and the other three calls should happen every time, unless by chance you've previously made a light curve within ~50 pixels of the star you're interested in. That seems to be the correct behavior.

The singular matrix issue is really a strange one since it's working fine for me! I get a success running the following.

s = eleanor.Source(gaia=4675352109658261376, sector=1)
data = eleanor.TargetData(s, height=15, width=15, bkg_size=31, do_psf=True, do_pca=True)

I have one question and then a few tests for you to try running to help us diagnose.

1) can you please post the full traceback of the error you're seeing, that ends with "singular matrix?" Seeing exactly what lines cause that to fail will point us towards what specific matrix we should be investigating.

2) If you pass through a TIC ID instead of a Gaia ID in your call to eleanor.Source(), do you see the same error? The TIC ID for this target is 38846515.

3) After the call to .Source (which works fine, yes?) can you please print star.pm_path and star.postcard_path and make sure they are the same as the download location on your machine? For your case that should be /home/detal/.eleanor/mastDownload/HLSP/hlsp_eleanor_tess_ffi_postcard-s0001-4-1-cal-0902-1078_tess_v2_pc/

That should be a start that will point us in the right direction to try to diagnose, I think.

[yaelnaze]

To afeinstein20 :

Are you getting the same issues as you posted before, when using the same call and after emptying the .eleanor directory?

yes, that's what worries me ! (example below is with .eleanor directory fully erased before doing anything)

To benmontet :

1. here it is

   data = eleanor.TargetData(star, height=15, width=15, bkg_size=31, do_psf=True, do_pca=True) WARNING: FITSFixedWarning: 'datfix' made the change 'Set MJD-OBS to 58324.812768 from DATE-OBS. Set DATE-BEG to '2018-07-25T19:31:32.353' from MJD-BEG'. [astropy.wcs.wcs] WARNING: The "exact" method is not yet implemented for rectangular apertures -- using "subpixel" method with "subpixels=32" [photutils.aperture.core] Traceback (most recent call last): File "", line 1, in File "/home/naze/.local/lib/python3.6/site-packages/eleanor/targetdata.py", line 213, in init self.corrected_flux(pca=True) File "/home/naze/.local/lib/python3.6/site-packages/eleanor/targetdata.py", line 1212, in corrected_flux lc_pred = calc_corr(f, cx, cy, skip) File "/home/naze/.local/lib/python3.6/site-packages/eleanor/targetdata.py", line 1203, in calc_corr x = xhat(cm, norm_l[skip:]) File "/home/naze/.local/lib/python3.6/site-packages/eleanor/targetdata.py", line 1144, in xhat ATAinv = np.linalg.inv(ATA) File "<__array_function__ internals>", line 6, in inv File "/home/naze/.local/lib/python3.6/site-packages/numpy/linalg/linalg.py", line 547, in inv ainv = _umath_linalg.inv(a, signature=signature, extobj=extobj) File "/home/naze/.local/lib/python3.6/site-packages/numpy/linalg/linalg.py", line 97, in _raise_linalgerror_singular raise LinAlgError("Singular matrix") numpy.linalg.LinAlgError: Singular matrix

2. yes, gaia or tic yield the same problem for targetdata - simply, using gaia sometimes also doesn't work at the eleanor.source stage...

3. it works fine usually (not always for gaia) ; postcard path is ok, the other one says

Traceback (most recent call last): File "", line 1, in AttributeError: 'Source' object has no attribute 'pm_path'

I really fail to understand, especially since lightkurve is working without problem - I can get anything I want...

[benmontet]

Okay thanks, the traceback is useful.

Can you download the latest version from github, and verify that you see the same issue with the most recent version? (If you've installed from pip previously, this will be a very slightly updated version beyond 1.0.1.) I think we might have the fix in this latest version.

After you clone the repository, if you are working in the main respository your python should automatically import this version. You can test that by typing eleanor.__path__ after you import eleanor and verifying that the location it returns is the location of your github directory and not your standard python install location.

If that does not work, can you repeat the same call, but including the argument tc=True in your call to eleanor.Source? That will help me isolate where the problem is.

[yaelnaze]

Hello,

Done (version is 1.0.2 now) but same problem as always...

Including tc=True changes the problem somewhat :

star = eleanor.Source(tic=38846515, sector=1, tc=True) Downloading URL https://mast.stsci.edu/tesscut/api/v0.1/astrocut?ra=68.9597092399964&dec=-64.0270368810381&y=31&x=31&units=px&sector=1 to /home/naze/.eleanor/tesscut/tesscut_20200120142419.zDownloading URL https://mast.stsci.edu/tesscut/api/v0.1/astrocut?ra=68.9597092399964&dec=-64.0270368810381&y=31&x=31&units=px&sector=1 to /home/naze/.eleanor/tesscut/tesscut_20200120142419.zDownloading URL [...] to /home/naze/.eleanor/tesscut/tesscut_20200120142419.zip ... [Done] Inflating...

print('Found TIC {0} (Gaia {1}), with TESS magnitude {2}, RA {3}, and Dec {4}'.format(star.tic, star.gaia, star.tess_mag, star.coords[0], star.coords[1])) Found TIC 38846515 (Gaia 4675352109658261376), with TESS magnitude 10.3157, RA 68.9597092399964, and Dec -64.0270368810381

data = eleanor.TargetData(star, height=15, width=15, bkg_size=31, do_psf=True, do_pca=True) Traceback (most recent call last): File "", line 1, in File "/home/naze/.local/lib/python3.6/site-packages/eleanor-1.0.2-py3.6.egg/eleanor/targetdata.py", line 173, in init self.ffiindex = self.post_obj.ffiindex File "/home/naze/.local/lib/python3.6/site-packages/eleanor-1.0.2-py3.6.egg/eleanor/postcard.py", line 351, in ffiindex A = np.loadtxt(eleanorpath + '/metadata/s{0:04d}/cadences_s{0:04d}.txt'.format(sector)) File "/home/naze/.local/lib/python3.6/site-packages/numpy/lib/npyio.py", line 981, in loadtxt fh = np.lib._datasource.open(fname, 'rt', encoding=encoding) File "/home/naze/.local/lib/python3.6/site-packages/numpy/lib/_datasource.py", line 269, in open return ds.open(path, mode, encoding=encoding, newline=newline) File "/home/naze/.local/lib/python3.6/site-packages/numpy/lib/_datasource.py", line 623, in open raise IOError("%s not found." % path) OSError: /home/naze/.eleanor/metadata/s0001/cadences_s0001.txt not found.

[benmontet]

Okay, I think this is the same issue as #152. That person was able to solve their issue by removing both their ~/.eleanor and ~/.astropy directories and starting from there. Apparently there's something (maybe an old pointing model?) that is in one of the older eleanor formats in the .astropy directory that v1.0.2 is getting confused on. Can you please delete both of those and try from there?

[ethankruse]

FWIW, I was getting the LinAlg error and traced it back to the CBV files not existing. Right now, in TargetData.get_cbvs, if the CBV file isn't found, it quietly replaces real CBVs with arrays of all 0's, which then caused the LinAlg failure. It might be worth raising an error of "You're trying to do PCA, but the CBVs can't be found" in there instead of using the arrays of zeros.

Why the CBVs didn't exist is complicated, but it boiled down to update() failing. It got as far the first time as creating the directory, but then failing to populate it with anything unless I did something special. I suspect the details of that have to do with our unique computing setup, so I didn't bother raising it as an issue here since I found the workaround. But, the fact that the directory existed meant that Update() was never called again, since it never actually checks all the needed files are in the directory, just that the directory exists.

Based on the errors in this issue, it looks like it's all boiling down to the files (CBVs at first, now cadences.txt) not being in the sector's metadata directory for whatever reason even though the metadata directory exists.

[benmontet]

Yes you should end up with 19 files in your eleanor/metadata/s0001 directory. Files called 'cadences.txt', 'quality.txt', 'target_s0001.fits', and 16 cbv_components_s0001_000x_000y.txt files where x and y range from 1 to 4. If going the test two above doesn't fix things (which it might, perhaps the issue is ~/.astropy keeping behind old calls to get some of this information in its cache?) can you let us know which files in that directory you're missing?

[yaelnaze]

I thus eliminated the .eleanor AND .astropy directories... but to no avail. using star = eleanor.Source(tic=38846515, sector=1) or star = eleanor.Source(gaia=4675352109658261376, sector=1) yields .... Downloading URL https://mast.stsci.edu/tesscut/api/v0.1/astrocut?ra=68.95970833333332&dec=-64.02703Downloading URL https://mast.stsci.edu/tesscut/api/v0.1/astrocut?ra=68.95970833333332&dec=-64.02703611111112&y=31&x=31&units=px&sector=1 to ./tesscut_20200127160150.zip ... [Done] Inflating... This is the first light curve you have made for this sector. Getting eleanor metadata products for Sector 1... This will only take a minute, and only needs to be done once. Any other light curves you make in this sector will be faster. ** Resuming transfer from byte position 2039040 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 213 100 213 0 0 541 0 --:--:-- --:--:-- --:--:-- 540 Traceback (most recent call last): File "", line 1, in File "/home/naze/.local/lib/python3.6/site-packages/eleanor-1.0.2-py3.6.egg/eleanor/source.py", line 221, in init Update(sector=self.sector) File "/home/naze/.local/lib/python3.6/site-packages/eleanor-1.0.2-py3.6.egg/eleanor/update.py", line 120, in init self.get_target() File "/home/naze/.local/lib/python3.6/site-packages/eleanor-1.0.2-py3.6.egg/eleanor/update.py", line 201, in get_target os.rename(fn, eleanorpath + '/metadata/s{0:04d}/target_s{0:04d}.fits'.format(self.sector, self.sector)) OSError: [Errno 18] Invalid cross-device link: 'tess2018206045859-s0001-0000000033877176-0120-s_lc.fits' -> '/home/naze/.eleanor/metadata/s0001/target_s0001.fits'

the eleanor/metadat/s0001 directory simply is empty.

Re-doing the star call a second time seems to work but then singular matrix again.

I don't understand why the task continues if info has not been downloaded : rather than having zeros in matrix, wouldn't it be easier to have a "download didn't work, try again" or something similar ("gateway time out") ?

In the meantime, lightkurve works perfectly well - I could download all data, make masks, take out background,... It's really weird.

Cheers,

[ethankruse]

Yep, that's the same error I got. Seems like eleanor.Update is going to fail whenever you try to install it in most cluster/remote environments because the home directories are often mounted on different file systems than the working directories, causing that error. My workaround solution was to require the first call for a new sector's light curves to be from the home directory, but I think Update() needs to be fixed to avoid this.

The solution is to directly download all the metadata files into the ~/.eleanor metadata directory. Right now, it appears eleanor.Update downloads them into whatever directory you were in when you called the function, and then attempts to move/rename them into the metadata directory, causing this error mode. An alternative and perhaps simpler code change solution might be to just use shutil.move() instead of os.rename() since that stackoverflow link and the documentation seems to say it will avoid this error in Python 3.

And yes, also adding in the error that CBV files don't exist rather than allowing all zeros to cause the singular matrix.

[benmontet]

Thanks JJ!

In your eleanor.Source() object, there is a variable called eleanorpath. When you run the script on the data drive, does that get set appropriately? If not, I imagine hardcoding that between the two calls might fix this issue. That would look something like

star = eleanor.Source(blah) star.eleanorpath = '/home/user/.eleanor' data = eleanor.TargetData(star, blah)

Of course, if you're set up and are happy to run things on your home directory and then just save the outputs to your larger directory there's no reason to mess with things that are working!

On Sat, Jul 25, 2020 at 7:55 AM jjhermes notifications@github.com wrote:

I had been struggling with a similar issue for the last several hours. Just a heads up: I couldn't solve things with the metadata deletion. Instead I solved the issue by running eleanor on the same mounted drive as my ~/.eleanor folder. For example, my ~/.eleanor folder is on /home/user/.eleanor but I was running a script using eleanor on my larger drive /data/TESS. Running eleanor instead on /home/user/TESS got things back on the rails.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/afeinstein20/eleanor/issues/142#issuecomment-663749743, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKA5YUJCP6FF3II737SO7LR5H7MRANCNFSM4KEYPYTQ .