TimeMachine returns 502

[arteg]

Hi. Thanks for the API!

I created an account and subscribed to "TimeMachine". I'm getting "Internal server error" when attempting to use the historical API:

https://timemachine.pirateweather.net/forecast/{key}/59.9375,30.308611,1621857600?units=si

I tried a bunch of different coordinates and time formats – to no avail.

[alexander0042]

Hi,

Thanks for pointing this out- I meant to write a better error message here, but completely forgot about it! The issue isn't with the date format, but rather with the date itself- it's too close to the present, and the weather data hasn't been updated yet.

The data I pull from is here: https://registry.opendata.aws/ecmwf-era5/, and it tends to lag by a couple months, which explains why you're getting there error for a date in May. Unfortunately, I can't easily fix this, but I have updated the code to return a much more helpful error message!

Thanks for pointing this out, I really appreciate the feedback!

[arteg]

@alexander0042 thanks for the fix!

Although I still get 502 on certain dates, for example 2020-04-01 (2020-03-31 works fine):

https://timemachine.pirateweather.net/forecast/{key}/59.9375,30.308611,1585688400

[Sibqo]

Hi @arteg i am also facing this issue of 502. @alexander0042 kindly guide us about 502 gate way

[alexander0042]

Thanks for the heads up here! Seems to be a time zone issue, and I'm fixing it now

[alexander0042]

Ok, this has been corrected! Combination of a time zone glitch and an issue with dates that were the last day of a month, since they required two separate calls to the archive. I've updated the back end to look for this case, and fixed a couple other issues pertaining to units and the rain/ snow login while I was in there.

Thanks again for pointing this out, and let me know if any other issues crop up!

[drmrbrewer]

@alexander0042 I'm still facing the same issue... you said above that:

the issue isn't with the date format, but rather with the date itself

But for me it's working with the date as a Unix timestamp (seconds) but not in string format:

Not OK: https://timemachine.pirateweather.net/forecast/{key}/47.209,13.763,2010-05-12T12:00:00 Not OK: https://timemachine.pirateweather.net/forecast/{key}/47.209,13.763,2010-05-12T12:00:00Z Not OK: https://timemachine.pirateweather.net/forecast/{key}/47.209,13.763,2010-05-12T12:00:00+0000 OK: https://timemachine.pirateweather.net/forecast/{key}/47.209,13.763,1273665600

Whilst I can of course supply the date as a Unix timestamp, I thought that the aim of this API was to be a drop-in replacement for darksky... and the darksky supports all of the above date formats:

Side-notes:

(a) the darksky API seems to have a reprieve, until the end of 2022 (b) the pirateweather time machine fetch is rather slow... not that I can really complain!

[alexander0042]

Thanks for noticing this! The forecast side of things definitely gets more of my attention, so it's easy to miss things, but this is an important part of it!

You're spot on here- much like Octave and MATLAB, my goal is to treat anything that doesn't work the same way as a bug, so this is worth fixing! Luckily, the Time Machine processing script is all Python anyway, so I should be able to add in string support pretty easily. I'll post an update here when it's active.

As for the retrieval speed, you'll have to take that one up with AWS. I can't store enough data in my EFS bucket to handle these requests, so it has to go to S3 to get the data. Luckily, it's in Zarr format, so can be read directly, but still isn't as fast as the forecast. However, depending on your use case, I might have better news. In the next week or so, I'll be switching on a pretty major update, which will let short term (thinking 1 week) data be retrieved using the forecast API. I've updated things to use Fargate for processing, which opened a few doors to make this happen. It still won't work for deep archival stuff, but for some applications, should be much faster.

[drmrbrewer]

Great, thanks for looking into this and I look forward to the fix for the date format.

Regarding retrieval speed of the historical data, I'd mainly be getting data that is further back than 1 week, so the speed improvement you mention won't impact me. However, retrieving historical data is such a minor part of what I'm doing (the vast majority requires forecast data, not historical data) so it's really not a big deal that it takes a few seconds rather than milliseconds.

[alexander0042]

One unnecessarily long regular expression later, I think I've fixed this! It now handles those three date formats, and converts back to the correct units. If you have a second to try out some cases, it would be great ☔

[drmrbrewer]

I think it's now hardcoded to 12 May 2010? Seems to return data for that day no matter what date string I pass in.

[alexander0042]

Didn't see that one coming- I left my test string in the code, so that'll definitely do it! Took that line out and made it live again

[drmrbrewer]

Ha! Seems a bit temperamental... I got it to return data once but more often than not I'm getting:

[alexander0042]

That's a weird error, since it's indicating a network issue, so I think this is related to another AWS outage that's ongoing. It's mostly impacting their virtual machines, so I don't have too much exposure, but some of the networking seems funky. https://status.aws.amazon.com/

[drmrbrewer]

OK thanks, I'll try later... am getting nothing else but that error right now :(

[drmrbrewer]

@alexander0042 seems to be fine now... I guess this issue can be closed again... though I opened another.