[Data Issue]: MX data link throws 403

[celsomilne]

When did this happen?

2023-05-12

What zones are affected?

MX

What is the problem?

Description

It appears that the link for the production data for MX is broken. The production source page does not appear to have its data available for download. Running tests on it returns a 403 error, as does clicking on the files on the page itself (i.e., visit the URL and click on the CSV file).

requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://www.cenace.gob.mx/SIM/VISTA/REPORTES/EnergiaGenLiqAgregada.aspx

Other available data

I am not familiar with the operation of the SIM, but I believe there are alternate sources of data:

• Demand data for the last 24 hours: See the graph here

Market Day Ahead (MDA) data

• Electrical power by load zone: See published data here
• Prices: Published data here

Real time prices

• Real time prices: here

Demand forecasts

• Generation forecasts: here
• Demand forecasts: here

Suggested fix

I am not sure which of these data sources are going to be the right ones, but I believe a solution will may include all / some of them! MDA data may be a good enough proxy for now, as might reconciling them with real time prices.

[VIKTORVAV99]

These are some great sources!

I just took a quick look at them but I think there are definitely something here we can use!

And yeah it does look like they are forbidden (403) and I hit that fot the last source you provided as well.

It seems like we can implement some of this data more or less right away (the generation forecasts for wind and solar) but others will need some investigation. The teams are quite busy at the time but I'll check in with them to see if this is enough to start estimating these zones once we have collected enough data.

[shuuji3]

Regarding the 403 error, this looks like due to their geo-blocking. I couldn't access the CSV file from Japan, but it can be downloaded from the Google Cloud server in the US data center. So if it's possible to run the data fetching on a specific location, the above CSV data source could be used as well.

[VIKTORVAV99]

Regarding the 403 error, this looks like due to their geo-blocking. I couldn't access the CSV file from Japan, but it can be downloaded from the Google Cloud server in the US data center. So if it's possible to run the data fetching on a specific location, the above CSV data source could be used as well.

I'll see if we can add this to your proxies and see what data we can get 🙂

[VIKTORVAV99]

Just double checked, it looks like the parser is unblocked for our parser servers so that is not an issue for the parser as far as I can tell.