Open havok2063 opened 5 years ago
havok2063
commented
5 years ago Looks like there is python module error when attempting to raise a MarvinError exception. The code itself may be hitting the standard API rate limit but fails when attempting to raise the proper error.
havok2063
commented
5 years ago Updated info and traceback from Nicola, which is an API rate limit error.
"I tried adding a sleep timer every 50 galaxies for 30s by adding
if m == 50:
time.sleep(30)within my 'for' loop after the 'if' condition (not within the 'if' condition). I realise this would only account for the first 50 galaxies anyway but should at least give me a larger number of galaxies (say, 100) to work with to know I am on the right track. But the code still only runs for 54 galaxies, give or take a few. Is the coding of this wrong? Apologies as this is the first time I have used Python - at uni I've always been taught Matlab. This is the full error I am now getting, which is slightly different to before:"
Traceback (most recent call last):
File "<ipython-input-1-484dab37e215>", line 1, in <module>
runfile('C:/Users/nicol/Documents/Project/MangaFilterAGN_Blue1.py', wdir='C:/Users/nicol/Documents/Project')
File "C:\Users\nicol\Documents\Spyder\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 668, in runfile
execfile(filename, namespace)
File "C:\Users\nicol\Documents\Spyder\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 108, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "C:/Users/nicol/Documents/Project/MangaFilterAGN_Blue1.py", line 175, in <module>
masks, fig, axes = maps.get_bpt(use_oi=False,show_plot=False)
File "C:\Users\nicol\Documents\Spyder\lib\site-packages\marvin\tools\maps.py", line 762, in get_bpt
use_oi=use_oi)
File "C:\Users\nicol\Documents\Spyder\lib\site-packages\marvin\utils\dap\bpt.py", line 269, in bpt_kewley06
nii = get_masked(maps, 'nii_6585', snr=get_snr(snr_min, 'nii'))
File "C:\Users\nicol\Documents\Spyder\lib\site-packages\marvin\utils\dap\bpt.py", line 51, in get_masked
gflux = maps['emline_gflux_' + emline]
File "C:\Users\nicol\Documents\Spyder\lib\site-packages\marvin\tools\maps.py", line 132, in __getitem__
return self.getMap(value)
File "C:\Users\nicol\Documents\Spyder\lib\site-packages\marvin\tools\maps.py", line 626, in getMap
return marvin.tools.quantities.Map.from_maps(self, best)
File "C:\Users\nicol\Documents\Spyder\lib\site-packages\marvin\tools\quantities\map.py", line 223, in from_maps
value, ivar, mask = cls._get_map_from_api(maps, prop)
File "C:\Users\nicol\Documents\Spyder\lib\site-packages\marvin\tools\quantities\map.py", line 321, in _get_map_from_api
'found a problem when getting the map: {0}'.format(str(ee)))
MarvinError: found a problem when getting the map: Requests Http Status Error: 429 Client Error: TOO MANY REQUESTS for url: https://dr15.sdss.org/marvin/api/maps/8486-6102/HYB10/GAU-MILESHC/map/emline_gflux/nii_6585/
Error accessing https://dr15.sdss.org/marvin/api/maps/8486-6102/HYB10/GAU-MILESHC/map/emline_gflux/nii_6585/: 429-Rate Limit Exceeded.
You can submit this error to Marvin GitHub Issues (https://github.com/sdss/marvin/issues/new).
Fill out a subject and some text describing the error that just occurred.
If able, copy and paste the full traceback information into the issue as well.
The task they are doing, “given a list of targets, looping over to access their BPT classifications, computing a ratio, and using that ratio to select a subsample of targets”.
More info from Nicola
"Given a list of targets, looping over all of them to access each of their BPT maps, making a classification without OI for each, then accessing the masks within each to count the number of spaxels classified as global within the agn mask.
I repeat this for the global spaxels within each of the other masks, excluding the invalid one to ignore background (essentially finding the number of AGN spaxels compared to the size of the galaxy for each).
If the ratio of (AGN spaxels)/(total occupied spaxels) is 5% or less, these are classified as non-AGN galaxies and stored in a blank array, otherwise, they are discarded. Outside the loop the colour-magnitude diagram of the remaining galaxies is then plotted."
Describe the bug By Nicola Russell. "I am currently using Marvin data in Python on a Windows machine. I am trying to obtain BPT diagrams (NII and SII only) over a selection of 1944 galaxies within a 'for' loop. However, after a certain number of iterations I keep getting the error message:
It means that instead of iterating over 1944 selected galaxies, it only iterates over 50 or so. The code works perfectly fine with a selection of only 20 galaxies with makes me think it is a storage issue somewhere rather than a coding issue."
To Reproduce "Here is (I think) the relevant snippet of the 'for' loop I'm using [BluePlates refers to the Galaxy IDs of the previous sample cut]:" Code that runs to produce the error.
Traceback is as follows.
Expected behavior The loop should complete or fail correctly when hitting the API rate limit
System
Additional context The data is initially being accessed remotely and then stored locally using os.environ. There is previous code making cuts to the sample size that use this remote data and store the relevant galaxy IDs in an array. The code I have written then applies only to the galaxies in that array so I suppose it is still accessing the data over your API which may cause the rate limit problem. The code tends to run for around 10-15 minutes before I get the error.