search

USACE / cumulus

Cumulus project issue tracking and project planning
MIT License
3 stars 2 forks source link

[BUG REPORT]: APRFC QPF grids dropped beginning ~1 Sept #424

Closed CDuncPOA closed 1 month ago

CDuncPOA commented 2 months ago

Alaska-Pacific RFC QPF available grids started decreasing around 1 Sept 2024. The APRFC told me that they did some network upgrades that changed some IP addresses and firewall settings at that time, which might be the root cause of the issue.

Screenshot of Cumulus product page (with days missing some grids circled in red):

image

Expected behavior:

Could the Cumulus team look into restoring the dropped QPF grids? And also double-check the QPE gridded products as well? Or let me know if this is an issue on the RFC end and I can follow up with them.

Desktop:

adamscarberry commented 2 months ago

We were using https://cbt.crohms.org/akgrids, but it looks like they restructured their site.

Hey @ArthurArmour Do you have any insight about the akgrids? I remember you had pointed us to this source a while back in an email thread.

cc @Enovotny

jbkolze commented 2 months ago

We were using https://cbt.crohms.org/akgrids, but it looks like they restructures their site.

Hey @ArthurArmour Do you have any insight about the akgrids? I remember you had pointed us to this source a while back in an email thread.

cc @Enovotny

Sorry, should've updated here -- @CDuncPOA is hanging out in LRL right now so I'm trying to help her out with this. We reached out to APRFC and are waiting to get an update from their end. Hopefully they just moved/reorganized things and we can just update the DAG to point in the right direction, but we'll know more once we hear back.

adamscarberry commented 2 months ago

Great, thanks @jbkolze

CDuncPOA commented 1 month ago

I think this is where we're at but @jbkolze can check my work: CENWP (which hosts my data) had to switch to a new virtual server in Amazon AWS Cloud. So APRFC has updated their known_hosts file, and it seems to be delivering grids on their end. The current problem is that the https://cbt.crohms.org/akgrids link that Cumulus scrapes for APRFC grids is now returning a 404 message. So we are trying to restore a HTTPS connection. And will probably need to get a new link from @ArthurArmour

FHanbali commented 1 month ago

Thanks for the update Rosie.

FHanbali commented 1 month ago

@ArthurArmour is there a direct grids source link from the APRFC office itself by any chance? Or is the APRFC using you CHROMS servers for data dissemination?

ArthurArmour commented 1 month ago

I am looking to see how we had things configured so the data could be accessed via cbt.crohms.org/akgrids. It is likely a configuration in httpd.conf that did not get copied over to the new server.

I faintly recall doing something in httpd.conf to define the location where akgrids is.

Looking into it.

Cheers, Art

From: Fauwaz Hanbali @.> Sent: Wednesday, October 2, 2024 2:13 PM To: USACE/cumulus @.> Cc: Armour, Arthur D CIV USARMY CENWP (USA) @.>; Mention @.> Subject: [Non-DoD Source] Re: [USACE/cumulus] [BUG REPORT]: APRFC QPF grids dropped beginning ~1 Sept (Issue #424)

@ArthurArmourBlockedhttps://github.com/ArthurArmour is there a direct grids source link from the APRFC office itself by any chance? Or is the APRFC using you CHROMS servers for data dissemination?

— Reply to this email directly, view it on GitHubBlockedhttps://github.com/USACE/cumulus/issues/424#issuecomment-2389705598, or unsubscribeBlockedhttps://github.com/notifications/unsubscribe-auth/AUBU6SKDMNPROBDYPCJ4Q43ZZROV7AVCNFSM6AAAAABN656XK6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBZG4YDKNJZHA. You are receiving this because you were mentioned.Message ID: @.**@.>>

ArthurArmour commented 1 month ago

I believe I found what is needed. With the switch to the new VM for our data exchange server, I lost some capabilities that I am trying to get back. Then I can make the change in the appropriate Apache-HTTPD conf file and access to https://cbt.crohms.org/akgrids should be restored.

Hopefully Thursday of this week.

Cheers, Art

From: CDuncPOA @.> Sent: Wednesday, October 2, 2024 1:59 PM To: USACE/cumulus @.> Cc: Armour, Arthur D CIV USARMY CENWP (USA) @.>; Mention @.> Subject: [Non-DoD Source] Re: [USACE/cumulus] [BUG REPORT]: APRFC QPF grids dropped beginning ~1 Sept (Issue #424)

I think this is where we're at but @jbkolzeBlockedhttps://github.com/jbkolze can check my work: CENWP (which hosts my data) had to switch to a new virtual server in Amazon AWS Cloud. So APRFC has updated their known_hosts file, and it seems to be delivering grids on their end. The current problem is that the https://cbt.crohms.org/akgridsBlockedhttps://cbt.crohms.org/akgrids link that Cumulus scrapes for APRFC grids is now returning a 404 message. So we are trying to restore a HTTPS connection. And will probably need to get a new link from @ArthurArmourBlockedhttps://github.com/ArthurArmour

— Reply to this email directly, view it on GitHubBlockedhttps://github.com/USACE/cumulus/issues/424#issuecomment-2389684914, or unsubscribeBlockedhttps://github.com/notifications/unsubscribe-auth/AUBU6SKE3A4WRUVR3T263BLZZRM7VAVCNFSM6AAAAABN656XK6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBZGY4DIOJRGQ. You are receiving this because you were mentioned.Message ID: @.**@.>>

ArthurArmour commented 1 month ago

The issue has been resolved. The cbt.crohms.org server has been updated so you can get to https://cbt.crohms.org/akgrids

Let me know if this takes care of things.

Cheers, Art

From: Armour, Arthur D CIV USARMY CENWP (USA) Sent: Wednesday, October 2, 2024 2:19 PM To: USACE/cumulus @.>; USACE/cumulus @.> Cc: Mention @.***> Subject: RE: [Non-DoD Source] Re: [USACE/cumulus] [BUG REPORT]: APRFC QPF grids dropped beginning ~1 Sept (Issue #424)

I am looking to see how we had things configured so the data could be accessed via cbt.crohms.org/akgrids. It is likely a configuration in httpd.conf that did not get copied over to the new server.

I faintly recall doing something in httpd.conf to define the location where akgrids is.

Looking into it.

Cheers, Art

From: Fauwaz Hanbali @.**@.>> Sent: Wednesday, October 2, 2024 2:13 PM To: USACE/cumulus @.**@.>> Cc: Armour, Arthur D CIV USARMY CENWP (USA) @.**@.>>; Mention @.**@.>> Subject: [Non-DoD Source] Re: [USACE/cumulus] [BUG REPORT]: APRFC QPF grids dropped beginning ~1 Sept (Issue #424)

@ArthurArmourBlockedhttps://github.com/ArthurArmour is there a direct grids source link from the APRFC office itself by any chance? Or is the APRFC using you CHROMS servers for data dissemination?

— Reply to this email directly, view it on GitHubBlockedhttps://github.com/USACE/cumulus/issues/424#issuecomment-2389705598, or unsubscribeBlockedhttps://github.com/notifications/unsubscribe-auth/AUBU6SKDMNPROBDYPCJ4Q43ZZROV7AVCNFSM6AAAAABN656XK6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBZG4YDKNJZHA. You are receiving this because you were mentioned.Message ID: @.**@.>>

FHanbali commented 1 month ago

Thanks @ArthurArmour, it looks like there's data posted there going back a couple of weeks or so.

@Enovotny it doesn't look like Cumulus/Airflow picked up on the restored connection, can you verify please?

jbkolze commented 1 month ago

@FHanbali @Enovotny I'm pretty sure the new server is just formatting the file list differently than the old one did. Probably just need to fix the BeautifulSoup parsing. I'll take a look.

jbkolze commented 1 month ago

@FHanbali Well, it actually looks like the parsing inspects the tags themselves so it started working without any tweaks. We're getting grids in prod now (and it also works in my local dev). I think we should be good to go, or at least are back to where we started before the server change. I think we're going to start trying to do a little integration with @CDuncPOA 's models so we'll open a new issue if anything else comes up.