search

scimma / blast

Django web app for the automatic characterization of supernova hosts
MIT License
1 stars 3 forks source link

[Bug] trim_transient_images task errors: buffer is too small for requested array #267

Closed manning-ncsa closed 6 days ago

manning-ncsa commented 1 week ago

I'm seeing a lot of these errors for the trim_transient_images task recently on the production instance:

[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871294717Z [2024-11-16 02:29:07,870: ERROR/ForkPoolWorker-572] Task host.system_tasks.trim_transient_images[aab2ae8a-6ead-4c98-bbfd-bd8496b47883] raised unexpected: TypeError('buffer is too small for requested array')
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871326898Z Traceback (most recent call last):
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871331947Z   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871335995Z     R = retval = fun(*args, **kwargs)
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871340153Z                  ^^^^^^^^^^^^^^^^^^^^
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871344962Z   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871348839Z     return self.run(*args, **kwargs)
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871352766Z            ^^^^^^^^^^^^^^^^^^^^^^^^^
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871357124Z   File "/app/host/system_tasks.py", line 303, in trim_transient_images
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871361112Z     TrimTransientImages().run_process()
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871368275Z   File "/app/host/system_tasks.py", line 282, in run_process
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871372563Z     trim_images(transient)
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871376781Z   File "/app/host/trim_images.py", line 20, in trim_images
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871380639Z     trim_image(c)
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871384726Z   File "/app/host/trim_images.py", line 82, in trim_image
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871388654Z     hdu.writeto(cutout.fits.name, overwrite=True)
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871394184Z   File "/usr/local/lib/python3.11/site-packages/astropy/io/fits/hdu/hdulist.py", line 1043, in writeto
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871416186Z     hdu._writeto(hdulist._file)
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871420384Z   File "/usr/local/lib/python3.11/site-packages/astropy/io/fits/hdu/base.py", line 722, in _writeto
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871424261Z     self._writeto_internal(fileobj, inplace, copy)
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871428169Z   File "/usr/local/lib/python3.11/site-packages/astropy/io/fits/hdu/base.py", line 728, in _writeto_internal
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871432407Z     data_offset, data_size = self._writedata(fileobj)
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871436464Z                              ^^^^^^^^^^^^^^^^^^^^^^^^
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871440271Z   File "/usr/local/lib/python3.11/site-packages/astropy/io/fits/hdu/base.py", line 675, in _writedata
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871444069Z     size += self._writedata_direct_copy(fileobj)
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871447936Z             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871452084Z   File "/usr/local/lib/python3.11/site-packages/astropy/io/fits/hdu/base.py", line 704, in _writedata_direct_copy
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871455941Z     raw = self._get_raw_data(self._data_size, "ubyte", self._data_offset)
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871459778Z           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871463625Z   File "/usr/local/lib/python3.11/site-packages/astropy/io/fits/hdu/base.py", line 559, in _get_raw_data
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871467422Z     return self._file.readarray(offset=offset, dtype=code, shape=shape)
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871471209Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871475087Z   File "/usr/local/lib/python3.11/site-packages/astropy/io/fits/file.py", line 403, in readarray
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871479184Z     return np.ndarray(
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871483242Z            ^^^^^^^^^^^
[pod/celery-worker-1/celery] 2024-11-16T02:29:07.871487109Z TypeError: buffer is too small for requested array
djones1040 commented 6 days ago

draft PR #268 fixes this I think by allowing images to be removed and re-downloaded during image trimming. Tests on prod seem to work.

djones1040 commented 6 days ago

think this should work -- merged into main