kcor.py key error

[nimatraore]

When running kcor.py in our Helioviewer instance at IAS, we have a key error:

For information, we are up to date with the Helioviewer and our sunpy version is 4.0.0.

Thank you for your help !

[dgarciabriseno]

Hi @nimatraore I'm trying to reproduce, can you reply with which commit of helioviewer you're on? Also please send the output of pip freeze so I can align my python environment with yours.

I'm on a clean master branch with sunpy 4.0.0 and it doesn't crash for me, so I'll need a bit more info to help. Thanks!

[nimatraore]

Hi @dgarciabriseno, For helioviewer/api, I'm on the commit d598957 And for helioviewer.org, I'm on the commit c93da2f

And beliow is the output of pip freeze:

aiohttp==3.8.1
aiosignal==1.2.0
astropy==5.0.4
async-timeout==4.0.2
attrs==20.3.0
beautifulsoup4==4.9.3
bokeh==2.4.2
Bottleneck==1.2.1
certifi==2020.6.20
chardet==4.0.0
charset-normalizer==2.0.12
cryptography==3.3.2
cycler==0.10.0
decorator==4.4.2
defusedxml==0.6.0
et-xmlfile==1.0.1
frozenlist==1.3.0
Glymur==0.9.9
html5lib==1.1
httplib2==0.18.1
idna==2.10
importlib-metadata==1.6.0
iniconfig==0.0.0
jdcal==1.0
Jinja2==2.11.3
joblib==0.17.0
kiwisolver==1.3.1
llvmlite==0.35.0
lxml==4.6.3
MarkupSafe==1.1.1
matplotlib==3.3.4
more-itertools==4.2.0
multidict==6.0.2
mysqlclient==1.4.4
numba==0.52.0
numexpr==2.7.2
numpy==1.19.5
odfpy==1.4.1
olefile==0.46
openpyxl==3.0.3
packaging==20.9
pandas==1.1.5
parfive==1.5.1
Pillow==8.1.2
pluggy==0.13.0
psutil==5.8.0
py==1.10.0
py-cpuinfo==5.0.0
pycurl==7.43.0.6
pyerfa==2.0.0.1
Pygments==2.7.1
PyGObject==3.38.0
pyparsing==2.4.7
PySimpleSOAP==1.16.2
pytest==6.0.2
python-apt==2.2.1
python-dateutil==2.8.1
python-debian==0.1.39
python-debianbts==3.1.0
pytz==2021.1
PyYAML==5.3.1
reportbug==7.10.3+deb11u1
requests==2.25.1
scipy==1.6.0
simplejson==3.17.2
six==1.16.0
soupsieve==2.2.1
sunpy==4.0.0
tables==3.6.1
toml==0.10.1
tornado==6.1
tqdm==4.64.0
typing-extensions==4.1.1
urllib3==1.26.5
webencodings==0.5.1
xlwt==1.3.0
yarl==1.7.2
zipp==1.0.0

[dgarciabriseno]

Hmm I still can't reproduce. The error is happening because the Helioviewer database expects that key to be KCor but you're getting the full title COSMO_K-Coronagraph This is read via sunpy, which was why I was thinking it was a problem with the version. But even running 4.0.0 I still get KCor and everything works ok.

Can you check out this commit 6f38077773f00001d81d08c09a65eec56efd9d81, I added some extra logging around your failure, and send me the output of this command: python downloader.py -d kcor -s "2022-05-30 18:39:00" -e "2022-05-30 18:40:00" This will run the kcor downloader on a tiny list of files that are failing in the traceback you sent. This will also save one of the maps to a file, so can you also send me <api>/install/helioviewer/sunpy_map.pickle after running it?

[wafels]

Nima

Could you try reading a KCor file into SunPy directly using the SunPy on your server?

Thanks,

Jack

On Thu, Jun 16, 2022 at 11:48 Daniel Garcia Briseno < @.***> wrote:

Hmm I still can't reproduce. The error is happening because the Helioviewer database expects that key to be KCor but you're getting the full title COSMO_K-Coronagraph This is read via sunpy, which was why I was thinking it was a problem with the version. But even running 4.0.0 I still get KCor and everything works ok.

Can you check out this commit 6f38077 https://github.com/Helioviewer-Project/api/commit/6f38077773f00001d81d08c09a65eec56efd9d81, I added some extra logging around your failure, and send me the output of this command: python downloader.py -d kcor -s "2022-05-30 18:39:00" -e "2022-05-30 18:40:00" This will run the kcor downloader on a tiny list of files that are failing in the traceback you sent. This will also save one of the maps to a file, so can you also send me

/install/helioviewer/sunpy_map.pickle after running it? — Reply to this email directly, view it on GitHub , or unsubscribe . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[dgarciabriseno]

the sunpy_map.pickle created by that test branch is the KCor map created by sunpy. I want to confirm that it looks the same as what's generated by our server.

[nimatraore]

Ah okay, the output of the command python downloader.py -d kcor -s "2022-05-30 18:39:00" -e "2022-05-30 18:40:00" gives this:

Initializing HVPull
Querying time range 2022-05-30 18:39:00 - 2022-05-30 18:40:00
(kcor) Scanning http://download.hao.ucar.edu/jp2/kcor/2022/05/30/white-light-pB
Time filtered file count: 0
Number of files from aia = 0
Time filtered file count: 0
Number of files from hmi = 0
Number of files available in time range from high volume instruments= 2
High volume instruments are = aia hmi
Stride value = 20
Number of strided files from high volume instruments = 0
Number of new URLS = 0
Found 2 new files
(kcor) Downloaded http://download.hao.ucar.edu/jp2/kcor/2022/05/30/white-light-pB/2022_05_30__18_39_05__MLSO_KCOR_KCOR_white-light-pB.jp2 (0.120 MB/s) [100.00%]
(kcor) Downloaded http://download.hao.ucar.edu/jp2/kcor/2022/05/30/white-light-pB/2022_05_30__18_39_20__MLSO_KCOR_KCOR_white-light-pB.jp2 (0.125 MB/s) [50.00%]
> /var/www/api/install/helioviewer/jp2parser.py(49)getData()
-> save_map(imageData)
(Pdb) c
Initial image data
nickname:
observatory: COSMO K-Coronagraph
instrument:  COSMO K-Coronagraph
detector:
Modified image data
nickname:
observatory: COSMO_K-Coronagraph
instrument:  COSMO
detector:
Removed /var/www/helioviewer/incoming/2022_05_30__18_39_20__MLSO_KCOR_KCOR_white-light-pB.jp2
Renamed /var/www/helioviewer/incoming/2022_05_30__18_39_20__MLSO_KCOR_KCOR_white-light-pB.jp2.tmp.jp2 to /var/www/helioviewer/incoming/2022_05_30__18_39_20__MLSO_KCOR_KCOR_white-light-pB.jp2
> /var/www/api/install/helioviewer/jp2parser.py(49)getData()
-> save_map(imageData)
(Pdb) c
Initial image data
nickname:
observatory: COSMO K-Coronagraph
instrument:  COSMO K-Coronagraph
detector:
Modified image data
nickname:
observatory: COSMO_K-Coronagraph
instrument:  COSMO
detector:
Removed /var/www/helioviewer/incoming/2022_05_30__18_39_05__MLSO_KCOR_KCOR_white-light-pB.jp2
Renamed /var/www/helioviewer/incoming/2022_05_30__18_39_05__MLSO_KCOR_KCOR_white-light-pB.jp2.tmp.jp2 to /var/www/helioviewer/incoming/2022_05_30__18_39_05__MLSO_KCOR_KCOR_white-light-pB.jp2
Traceback (most recent call last):
  File "downloader.py", line 189, in <module>
    sys.exit(main())
  File "downloader.py", line 68, in main
    daemon.start(args.start, args.end, args.backfill)
  File "/var/www/api/install/helioviewer/hvpull/net/daemon.py", line 145, in start
    self.query(starttime, endtime)
  File "/var/www/api/install/helioviewer/hvpull/net/daemon.py", line 335, in query
    self.acquire(new_urls)
  File "/var/www/api/install/helioviewer/hvpull/net/daemon.py", line 470, in acquire
    self.ingest(finished)
  File "/var/www/api/install/helioviewer/hvpull/net/daemon.py", line 560, in ingest
    process_jp2_images(images, self.image_archive, self._db, self._cursor, True, None, self._cursor_v2)
  File "/var/www/api/install/helioviewer/jp2.py", line 52, in process_jp2_images
    insert_images(subset, sources, root_dir, db, cursor, mysql, step_fxn, cursor_v2)
  File "/var/www/api/install/helioviewer/jp2.py", line 102, in insert_images
    source = source[str(img[leaf])]
KeyError: 'COSMO_K-Coronagraph'

[nimatraore]

And here is the sunpy_map.pickle file

[dgarciabriseno]

Oops, pdb was supposed to just be there for me, forgot to delete it.

It looks like that map is coming back as a GenericMap instead of a KCorMap This is how it looks on my system.

nickname:    KCor
observatory: MLSO
instrument:  COSMO K-Coronagraph
detector:    KCor

I should have all the info from your pip freeze, but can you run import sunpy; sunpy.system_info() to confirm version info?

[nimatraore]

Oh actually you are right, we have two versions of sunpy because of python versions (sorry). Command with python was running with python2 version.

So when I run import sunpy; sunpy.system_info() with python, I have this:

Python 2.7.18 (default, Jul 14 2021, 08:11:37)
[GCC 10.2.1 20210110] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sunpy
>>> sunpy.system_info()
==============================
SunPy Installation Information
==============================

#######
General
#######
Time : Thursday, 16. June 2022 09:09PM UT
System : Linux
Processor :
Arch : 64bit
SunPy : 0.9.10
SunPy_git : a0d06b8310184cbc5d60289f1a2c89e3faafd375
OS: debian 11.3  (Linux 5.10.0-13-amd64 )

##################
Required Libraries
##################
Python: 2.7.18
NumPy: 1.16.6
SciPy: 1.2.2
matplotlib: 2.2.5
Astropy: 2.0.16
Pandas: 0.23.4

#####################
Recommended Libraries
#####################
###########
 Recommended Libraries
###########
beautifulsoup: 4.8.2
PyQt: NOT INSTALLED
SUDS: 0.6
Sqlalchemy: 1.3.20
Requests: 2.25.0

And with python3, I have this:

Python 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sunpy
>>> sunpy.system_info()
==============================
sunpy Installation Information
==============================

General
#######
OS: Debian GNU/Linux (11, Linux 5.10.0-13-amd64)
Arch: 64bit, ()
sunpy: 4.0.0
Installation path: /usr/local/lib/python3.9/dist-packages/sunpy-4.0.0.dist-info

Required Dependencies
#####################
astropy: 5.0.4
numpy: 1.19.5
packaging: 20.9
parfive: 1.5.1

Optional Dependencies
#####################
asdf: Missing asdf>=2.8.0; extra == "asdf"
asdf-astropy: Missing asdf-astropy>=0.1.1; extra == "asdf"
beautifulsoup4: 4.9.3
cdflib: Missing cdflib!=0.4.0,>=0.3.19; extra == "timeseries"
dask: Missing dask[array]>=2.0.0; extra == "dask"
drms: Missing drms>=0.6.1; extra == "net"
glymur: 0.9.9
h5netcdf: Missing h5netcdf>=0.8.1; extra == "timeseries"
h5py: Missing h5py>=3.1.0; extra == "timeseries"
matplotlib: 3.3.4
mpl-animators: Missing mpl-animators>=1.0.0; extra == "map" or "visualization"
pandas: 1.1.5
python-dateutil: 2.8.1
reproject: Missing reproject; extra == "docs" or "map"
scikit-image: Missing scikit-image>=0.16.0; extra == "image"
scipy: 1.6.0
sqlalchemy: Missing sqlalchemy>=1.3.4; extra == "database"
tqdm: 4.64.0
zeep: Missing zeep>=3.4.0; extra == "net"

[nimatraore]

And now I find like you when I run with python3 the command python3 downloader.py -d kcor -s "2022-05-30 18:39:00" -e "2022-05-30 18:40:00"

it gives this:

/var/www/api/install/helioviewer/jp2.py:123: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if step_function and (i + 1) % __STEP_FXN_THROTTLE__ is 0:
/var/www/api/install/downloader.py:86: DeprecationWarning: This method will be removed in future versions.  Use 'parser.read_file()' instead.
  config.readfp(open(default_userconfig))
Creating browser with me
Initializing HVPull
Querying time range 2022-05-30 18:39:00 - 2022-05-30 18:40:00
(kcor) Scanning http://download.hao.ucar.edu/jp2/kcor/2022/05/30/white-light-pB
Time filtered file count: 0
Number of files from aia = 0
Time filtered file count: 0
Number of files from hmi = 0
Number of files available in time range from high volume instruments= 2
High volume instruments are = aia hmi
Stride value = 20
Number of strided files from high volume instruments = 0
Number of new URLS = 0
Found 2 new files
(kcor) Downloaded http://download.hao.ucar.edu/jp2/kcor/2022/05/30/white-light-pB/2022_05_30__18_39_05__MLSO_KCOR_KCOR_white-light-pB.jp2 (0.120 MB/s) [100.00%]
(kcor) Downloaded http://download.hao.ucar.edu/jp2/kcor/2022/05/30/white-light-pB/2022_05_30__18_39_20__MLSO_KCOR_KCOR_white-light-pB.jp2 (0.126 MB/s) [50.00%]
> /var/www/api/install/helioviewer/jp2parser.py(49)getData()
-> save_map(imageData)
(Pdb) c
Initial image data
nickname:    KCor
observatory: MLSO
instrument:  COSMO K-Coronagraph
detector:    KCor
Modified image data
nickname:    KCor
observatory: MLSO
instrument:  COSMO
detector:    KCor
Removed /var/www/helioviewer/incoming/2022_05_30__18_39_20__MLSO_KCOR_KCOR_white-light-pB.jp2
Renamed /var/www/helioviewer/incoming/2022_05_30__18_39_20__MLSO_KCOR_KCOR_white-light-pB.jp2.tmp.jp2 to /var/www/helioviewer/incoming/2022_05_30__18_39_20__MLSO_KCOR_KCOR_white-light-pB.jp2
> /var/www/api/install/helioviewer/jp2parser.py(49)getData()
-> save_map(imageData)
(Pdb) c
Initial image data
nickname:    KCor
observatory: MLSO
instrument:  COSMO K-Coronagraph
detector:    KCor
Modified image data
nickname:    KCor
observatory: MLSO
instrument:  COSMO
detector:    KCor
Removed /var/www/helioviewer/incoming/2022_05_30__18_39_05__MLSO_KCOR_KCOR_white-light-pB.jp2
Renamed /var/www/helioviewer/incoming/2022_05_30__18_39_05__MLSO_KCOR_KCOR_white-light-pB.jp2.tmp.jp2 to /var/www/helioviewer/incoming/2022_05_30__18_39_05__MLSO_KCOR_KCOR_white-light-pB.jp2
Traceback (most recent call last):
  File "/var/www/api/install/downloader.py", line 189, in <module>
    sys.exit(main())
  File "/var/www/api/install/downloader.py", line 68, in main
    daemon.start(args.start, args.end, args.backfill)
  File "/var/www/api/install/helioviewer/hvpull/net/daemon.py", line 145, in start
    self.query(starttime, endtime)
  File "/var/www/api/install/helioviewer/hvpull/net/daemon.py", line 335, in query
    self.acquire(new_urls)
  File "/var/www/api/install/helioviewer/hvpull/net/daemon.py", line 470, in acquire
    self.ingest(finished)
  File "/var/www/api/install/helioviewer/hvpull/net/daemon.py", line 560, in ingest
    process_jp2_images(images, self.image_archive, self._db, self._cursor, True, None, self._cursor_v2)
  File "/var/www/api/install/helioviewer/jp2.py", line 52, in process_jp2_images
    insert_images(subset, sources, root_dir, db, cursor, mysql, step_fxn, cursor_v2)
  File "/var/www/api/install/helioviewer/jp2.py", line 102, in insert_images
    source = source[str(img[leaf])]
KeyError: 'MLSO'

And here is the new sunpy_map.pickle file which is generated with the python3

[dgarciabriseno]

Ah that explains it.

This new error looks makes it seem like MLSO isn't in your database. Check your datasources table for this, and insert it if it's not there (id does not have to be 83)

+----+------------+-------------+-------+---------------+---------+---------------+--------------+----------+----------+------------+
| id | name       | description | units | layeringOrder | enabled | sourceIdGroup | displayOrder | groupOne | groupTwo | groupThree |
+----+------------+-------------+-------+---------------+---------+---------------+--------------+----------+----------+------------+
| 83 | COSMO KCor | COSMO KCor  | NM    |             1 |       1 |               |            1 |        0 |        0 |          0 |
+----+------------+-------------+-------+---------------+---------+---------------+--------------+----------+----------+------------+

Then datasource_property should have these 4 rows. again sourceId doesn't need to be 83, but should align with whatever id is in the datasources table

+----------+-------------+-------+----------+-------------+---------+
| sourceId | label       | name  | fitsName | description | uiOrder |
+----------+-------------+-------+----------+-------------+---------+
|       83 | Observatory | MLSO  | MLSO     | MLSO        |       1 |
|       83 | Instrument  | COSMO | COSMO    | COSMO       |       2 |
|       83 | Detector    | KCor  | KCor     | KCor        |       3 |
|       83 | Measurement | 735   | 735      | 735         |       4 |
+----------+-------------+-------+----------+-------------+---------+

[nimatraore]

Oh yes indeed MLSO was not in our database. I just inserted the rows you mentionned into the DB.

The program ran fine this time:

/var/www/api/install/downloader.py:86: DeprecationWarning: This method will be removed in future versions.  Use 'parser.read_file()' instead.
  config.readfp(open(default_userconfig))
Creating browser with me
Initializing HVPull
Querying time range 2022-05-30 18:39:00 - 2022-05-30 18:40:00
(kcor) Scanning http://download.hao.ucar.edu/jp2/kcor/2022/05/30/white-light-pB
Time filtered file count: 0
Number of files from aia = 0
Time filtered file count: 0
Number of files from hmi = 0
Number of files available in time range from high volume instruments= 2
High volume instruments are = aia hmi
Stride value = 20
Number of strided files from high volume instruments = 0
Number of new URLS = 0
Found 2 new files
(kcor) Downloaded http://download.hao.ucar.edu/jp2/kcor/2022/05/30/white-light-pB/2022_05_30__18_39_05__MLSO_KCOR_KCOR_white-light-pB.jp2 (0.114 MB/s) [100.00%]
(kcor) Downloaded http://download.hao.ucar.edu/jp2/kcor/2022/05/30/white-light-pB/2022_05_30__18_39_20__MLSO_KCOR_KCOR_white-light-pB.jp2 (0.119 MB/s) [50.00%]
> /var/www/api/install/helioviewer/jp2parser.py(49)getData()
-> save_map(imageData)
(Pdb) c
Initial image data
nickname:    KCor
observatory: MLSO
instrument:  COSMO K-Coronagraph
detector:    KCor
Modified image data
nickname:    KCor
observatory: MLSO
instrument:  COSMO
detector:    KCor
Removed /var/www/helioviewer/incoming/2022_05_30__18_39_20__MLSO_KCOR_KCOR_white-light-pB.jp2
Renamed /var/www/helioviewer/incoming/2022_05_30__18_39_20__MLSO_KCOR_KCOR_white-light-pB.jp2.tmp.jp2 to /var/www/helioviewer/incoming/2022_05_30__18_39_20__MLSO_KCOR_KCOR_white-light-pB.jp2
> /var/www/api/install/helioviewer/jp2parser.py(49)getData()
-> save_map(imageData)
(Pdb) c
Initial image data
nickname:    KCor
observatory: MLSO
instrument:  COSMO K-Coronagraph
detector:    KCor
Modified image data
nickname:    KCor
observatory: MLSO
instrument:  COSMO
detector:    KCor
Removed /var/www/helioviewer/incoming/2022_05_30__18_39_05__MLSO_KCOR_KCOR_white-light-pB.jp2
Renamed /var/www/helioviewer/incoming/2022_05_30__18_39_05__MLSO_KCOR_KCOR_white-light-pB.jp2.tmp.jp2 to /var/www/helioviewer/incoming/2022_05_30__18_39_05__MLSO_KCOR_KCOR_white-light-pB.jp2
Added 2 images to database
Sleeping for 30 minutes.

Finished processing all files in requested time range
Exiting HVPull

And the sunpy_map.pickle file which is generated with.

Thank you very much !

[dgarciabriseno]

Looks good! you can go back to using the main branch now. Closing as resolved.