search

pietern / goestools

Tools to work with signals and files from GOES satellites
https://pietern.github.io/goestools/
BSD 2-Clause "Simplified" License
368 stars 82 forks source link

Unhandled key in ancillary text "grid_mapping" #76

Closed kidshare closed 4 years ago

kidshare commented 4 years ago

What is this error, it seems to happen when writing a file. Last successful write was 17:30 UTC April 2, 2020

terminate called after throwing an instance of 'std::runtime_error' what(): Failure at /home/mydisk/goestools/src/goesproc/handler_goesr.cc:138, Unhandled key in anciallry text "grid_mapping" Aborted (core dumped)

Is this problem coming down from the satellite or is it local?

nigelvh commented 4 years ago

Came to report this issue as well. Exact same error message, first crashed for me around 8:58 AM Pacific (15:58 UTC). Nothing has changed in my setup. I suspect new/changed info from the sat.

kidshare commented 4 years ago

Came to report this issue as well. Exact same error message, first crashed for me around 8:58 AM Pacific (15:58 UTC). Nothing has changed in my setup. I suspect new/changed info from the sat.

OK that confirms it! Thanks. I guess wait for them to correct it.

nigelvh commented 4 years ago

Edited line 138 to include the value field to see what it contained.

Key: grid_mapping Value: goes_imager_projection

I suspect this key can (at least in the short term) be ignored. Will test.

nigelvh commented 4 years ago

Looks like there are several new keys. Added code to ignore grid_mapping, and now getting the same error about a perspective_point_height key. Will look a bit more.

pietern commented 4 years ago

Added a commit to ignore the newly added keys: ca5a7e7ccc3af5dbcc6f19a32bd1c84b4a5b5c5e

It should work again after updating your installation to master.

I am not able to test if this works myself, so I'd appreciate confirmation whether or not this works.

kidshare commented 4 years ago

Added a commit to ignore the newly added keys: ca5a7e7

It should work again after updating your installation to master.

I am not able to test if this works myself, so I'd appreciate confirmation whether or not this works.

stalled again at same spot with same error message :(

nigelvh commented 4 years ago

@kidshare I suspect you did a make without a make install, or similar. I've just tested and the change works for me. Looking at the json there are quite a few new fields they added.

Old AncillaryText:

"AncillaryText":{
    "Channel":"13",
    "Imaging Mode":"ABI Mode 6",
    "Instrument":"FM2",
    "Region":"Mesoscale",
    "Resolution":"2km at nadir",
    "Satellite":"G17",
    "Segmented":"no",
    "Time of frame start":"2020-04-02T15:36:55.5Z"
    }

New AncillaryText:

"AncillaryText":{
    "Channel":"13",
    "Imaging Mode":"ABI Mode 6",
    "Instrument":"FM2",
    "Region":"Mesoscale",
    "Resolution":"2km at nadir",
    "Satellite":"G17",
    "Segmented":"no",
    "Time of frame start":"2020-04-02T17:54:25.5Z",
    "grid_mapping":"goes_imager_projection",
    "inverse_flattening":"298.257222 meter",
    "latitude_of_projection_origin":"0.0 degree",
    "longitude_of_projection_origin":"-137.0 degree",
    "perspective_point_height":"35786023.000000 meter",
    "semi_major_axis":"6378137.000000 meter",
    "semi_minor_axis":"6356752.314140 meter",
    "sweep_angle_axis":"x",
    "x_add_offset":"0.024668001 radian",
    "x_image_bounds":"0.024639999, 0.052639998 radian",
    "x_scale_factor":"0.000056000 radian",
    "y_add_offset":"0.120371997 radian",
    "y_image_bounds":"0.120399997, 0.092399999 radian",
    "y_scale_factor":"-0.000056000 radian"
    }
kidshare commented 4 years ago

Yes I think you're right. I see the changes in the handler file but they didn't make properly, trying again

kidshare commented 4 years ago

This is frustrating, I have rebuilt this from scratch and I get the same error

nigelvh commented 4 years ago

@kidshare The change removes the means to generate that error, so either you're compiling old code (though I believe you said you saw the updated changes in the file handler), or you're compiling the new data, but still using the old executables. On linux the 'which' command can point you to the path it's using. In my case:

nigel@goes:/media/goes/images$ which goesproc
/usr/local/bin/goesproc

Make sure your version of that file is being overwritten with the new compiled one. Yours may be in a different place than mine.

kidshare commented 4 years ago

Got it! I deleted the copies in /usr/bin (I did not think to look there)

lxe commented 4 years ago

Thanks for fixing this so fast.

autotunafish commented 4 years ago

šŸ‘ I have two goes rigs going and with them both failing simultaneously pointed me towards something about the downlink which seems a little presumptuous compared to a simple computer issue and it was just as easy as dragging goestools out of the home into the trash and reinstalling, outstanding! Thank youšŸ˜

ghost commented 4 years ago

Iā€™m running goestools on Ubuntu 18. Looks like Iā€™m getting images from sensors 7-15 but no fd/fc, getting fc from m1 and m2.

Recompiled to latest version on git, itā€™s working but Iā€™m getting these errors and still no fc.

VC 8: New S_PDU for 262, but didnā€™t finish previous one

VC 8: finished S_PDU for 262

VC 7: Zero length S_PDU for APID 229

VC 14: Zero length S_PDU for APID 454

VC 15: New S_PDU for 486, but didnā€™t finish previous one

VC 15: finished S_PDU for 486

My vit is in the ~130 range and no dropped packets? Is this an issue anyone else is having or just me?

autotunafish commented 4 years ago

I just restarted mine a few minutes ago currently in the middle of an M1 M2 swath, will update

-Yes, m1 and m2 appear more frequently about every 15 mins and seems that GOES 17 is the same, perhaps a frequency re-packing

Be patient

ghost commented 4 years ago

Much appreciated. I did a hard reset on my entire receiving computer again so weā€™ll see if that does anything, doubtful though.

ghost commented 4 years ago

Still getting the same errors with no fc. Wind knocked my antenna out of alignment and itā€™s getting a little late to be messing with it in the cold.

I can try to reinstall tomorrow but it seems to be a decoder problem as other images are coming in.

kidshare commented 4 years ago

I havent received any FC images either, something's wrong with ch02. The rest of the channels seem OK.

autotunafish commented 4 years ago

It sounds like your system is fine and that something changed with NOAA, here's their little bulletin board though I'm not sure if any of this is actionable here specifically https://www.ospo.noaa.gov/Operations/messages.html

ghost commented 4 years ago

Yeah, now that you mention it channel 2 is gone for me as well @kidshare.

Apologies for not knowing much on the subject, this is my first day of setting up my GOES station. Sorry for bringing the bad luck lol.

kidshare commented 4 years ago

Oh man your first day!

Ch 2 is used for the brightness scale and Ch 13 the color scale for the FC images.

ghost commented 4 years ago

Hopefully whateverā€™s going on gets fixed soon. But man, being my first day just seeing the channel 13 image and m1/m2 FCs made my day.

Thanks for the explanation!

kidshare commented 4 years ago

I'm staring at the VCID, looks like Ch02 is not coming down at all. Don't know what Ch63 is, it is not in the documentation.

ghost commented 4 years ago

Nothing has showed up yet on the bulletin @autotunafish posted. You thinking itā€™s a sat problem?

kidshare commented 4 years ago

Nothing has showed up yet on the bulletin @autotunafish posted. You thinking itā€™s a sat problem?

looks like it, I have not seen anything from ch02 in the past 45 minutes

autotunafish commented 4 years ago

I'm running 2 completely seperate goes 16 and 17 set ups and thet terminated at exactly the same time, yes it's a broadcast change, whether it's permanent or has anything to do with that schedule I'm not sure, I picked up a just few images from Goes 15 once on my 16 rig and recently also some regular Goes 17 which I did not pick up when I started but instead picked up way more NWS maps and txt files so they do definitely change things around sometimes

That goes 15 image pickup was around the time they stored goes 15 so I figured it might have had something to do with that wanting to test it before they turned it off or something

(Anyways it looks like the keys and values being ignored indicate some type of (apparently new?) image format goes_image_projection or perhaps the same renderer with a new header for something idk perhaps for geocolor?)

ghost commented 4 years ago

Quick update, got everything set up and received channel 2 and fc from GOES 16. Looks like most of those errors are gone and everything is working correctly. Will update if ch 02 goes blank again, but for now it looks like this was a temporary thing.

Thanks for the help everyone!

TP_PDU Drops are still there as others are mentioning.

creinemann commented 4 years ago

I am getting some unusual TP_PDU drops on these APID's: I looked them up in Volume 2 of the GOES Product Guide APID 294 No Entry APID 454 Current Transfer Gains Data APID 486 Channel 06 APID 642 No Entry APID 707 No Entry APID 807 No Entry APID 834 LPU Engineering Data APID1025 DPU Engineering Packet Data

Anyone else?

cyborgcnc commented 4 years ago

Same on this end with the TP_PDU drops (GOES 16)...on 807, 454 and 486....Like other have mentioned, FD/FC came back this morning, and also receiving the other channels...the latest update fixed the grid_mapping error for sure....we shall see....but I am glad I am receiving again! Other interesting thing, and I wonder if anyone has observed this, it seems that FC file sizes are smaller. I got one (aprox 13:15 EST) FC at the usual large Full res size (about 14MB) but the ones after, all are coming in at about 500K...again, I guess we shall see....

creinemann commented 4 years ago

hmmm, Just check my FC all are coming in at coming in a 10-14 MB full res size. I did see that my goesproc-goesr.conf file went back to 'default' after the patch, so I had to add my custom lines back into the new one.

autotunafish commented 4 years ago

Im showing normal here, The change was for almost exactly 24 hrs between img 20200402T143020Z and 20200403T140020Z

pietern commented 4 years ago

@creinemann If you updated your installation with make install, then the example configuration will have been clobbered and overwritten with the default one. If you keep a separate customized configuration, consider storing it some place else.

Glad to hear the FCs came back. Must have been a change to the stream assembly.

creinemann commented 4 years ago

Thanks Pieter, yes I found out about the conf file, and had a backup of my custom one. Any thoughts on if the new apid's are decodeable? They look like house keeping data...

Carl

On Sun, Apr 5, 2020, 2:32 PM Pieter Noordhuis notifications@github.com wrote:

@creinemann https://github.com/creinemann If you updated your installation with make install, then the example configuration will have been clobbered and overwritten with the default one. If you keep a separate customized configuration, consider storing it some place else.

Glad to hear the FCs came back. Must have been a change to the stream assembly.

ā€” You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pietern/goestools/issues/76#issuecomment-609469865, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALGT3Q4Y3HWIOGXMKHIZ7NDRLDMEHANCNFSM4L2VCT3Q .

creinemann commented 4 years ago

Getting these APID's

I looked them up in Volume 2 of the GOES Product Guide APID 294 No Entry APID 454 Current Transfer Gains Data APID 486 Channel 06 APID 642 No Entry APID 707 No Entry APID 807 No Entry APID 834 LPU Engineering Packet Data APID1025 DPU Engineering Packet Data

On Sun, Apr 5, 2020, 2:32 PM Pieter Noordhuis notifications@github.com wrote:

@creinemann https://github.com/creinemann If you updated your installation with make install, then the example configuration will have been clobbered and overwritten with the default one. If you keep a separate customized configuration, consider storing it some place else.

Glad to hear the FCs came back. Must have been a change to the stream assembly.

ā€” You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pietern/goestools/issues/76#issuecomment-609469865, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALGT3Q4Y3HWIOGXMKHIZ7NDRLDMEHANCNFSM4L2VCT3Q .

pietern commented 4 years ago

I don't recall APIDs being associated with particular products, only the VCs (virtual channels).

Can you link to the guide where you found this?

creinemann commented 4 years ago

Here is the GOES product guide VOL 2. I have requested a list of the updated APID but have not heard back as of yet. https://www.goes-r.gov/users/docs/PUG-L0-vol2.pdf

On Mon, Apr 6, 2020, 8:49 AM Pieter Noordhuis notifications@github.com wrote:

I don't recall APIDs being associated with particular products, only the VCs (virtual channels).

Can you link to the guide where you found this?

ā€” You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pietern/goestools/issues/76#issuecomment-609806415, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALGT3Q37T7YW55QFIHLCFQLRLHMXNANCNFSM4L2VCT3Q .

pietern commented 4 years ago

The APID list looks specific to the GRB data stream. I believe the HRIT APIDs are unrelated.

Since the original issue has been fixed, I'm closing the issue.