search

davidcarslaw / openair

Tools for air quality data analysis
https://davidcarslaw.github.io/openair/
GNU General Public License v2.0
305 stars 113 forks source link

Error running importKCL() function to download LondonAir monitoring data #386

Closed amystidworthy closed 4 months ago

amystidworthy commented 4 months ago

Bug description

The importKCL function is returning the following error:

Error in load(con) : cannot open the connection to 'http://www.londonair.org.uk/r_data/sites.RData' In addition: Warning message: In load(con) : URL 'https://www.londonair.org.uk/r_data/sites.RData': status was 'SSL connect error'

The importUKAQ function and similar functions are working normally. Is there a problem with the importKCL function?

Steps to reproduce

You can reproduce the error using this code:

test.data=importKCL(site="bt4",year=2024)

Openair version

2.18.2

davidcarslaw commented 4 months ago

Thanks Amy. I will need to check with the folks at Imperial where the data are stored. Note that a lot of local networks (including LondonAir) are available through the importLocal function - see here (section 3.2.2).

So,

test.data=importLocal(site="bt4",year=2024)
test.data
# A tibble: 4,535 × 9
   source site         code  date                  nox   no2    no    o3  pm10
   <chr>  <chr>        <chr> <dttm>              <dbl> <dbl> <dbl> <dbl> <dbl>
 1 local  Brent - Ikea BT4   2024-01-01 00:00:00  20.6  12.2   5.5    NA    27
 2 local  Brent - Ikea BT4   2024-01-01 01:00:00  21    12.8   5.3    NA    11
 3 local  Brent - Ikea BT4   2024-01-01 02:00:00  22.8  14     5.7    NA    11
 4 local  Brent - Ikea BT4   2024-01-01 03:00:00  13     8.5   2.9    NA     9
 5 local  Brent - Ikea BT4   2024-01-01 04:00:00  12.1   8.3   2.5    NA    11
 6 local  Brent - Ikea BT4   2024-01-01 05:00:00  17.9  12.2   3.8    NA    14
 7 local  Brent - Ikea BT4   2024-01-01 06:00:00  18.4  12.5   3.8    NA    19
 8 local  Brent - Ikea BT4   2024-01-01 07:00:00  22.9  16.3   4.3    NA    15
 9 local  Brent - Ikea BT4   2024-01-01 08:00:00  24.3  17.3   4.5    NA    20
10 local  Brent - Ikea BT4   2024-01-01 09:00:00  41.8  27.8   9.1    NA    23

Does that help?

Thanks David

jack-davison commented 4 months ago

This appears to be fixed.

openair::importKCL(site="bt4",year=2024)
#> Warning in openair::importKCL(site = "bt4", year = 2024): Some of the more
#> recent data may not be ratified.
#> NOTE - mass units are used 
#> ug/m3 for NOx, NO2, SO2, O3; mg/m3 for CO
#> PM10_raw is raw data multiplied by 1.3
#> # A tibble: 4,640 × 9
#>    date                  nox   no2    o3 pm10_raw  pm10  pm25 site         code 
#>    <dttm>              <dbl> <dbl> <dbl>    <dbl> <dbl> <dbl> <fct>        <chr>
#>  1 2024-01-01 00:00:00  20.5 12.1     NA     36.4 27.1     21 Brent - Ikea BT4  
#>  2 2024-01-01 01:00:00  20.9 12.8     NA     14.3 10.6     10 Brent - Ikea BT4  
#>  3 2024-01-01 02:00:00  22.7 14.0     NA     14.3 10.6     10 Brent - Ikea BT4  
#>  4 2024-01-01 03:00:00  13.0  8.51    NA     11.7  8.70    10 Brent - Ikea BT4  
#>  5 2024-01-01 04:00:00  12.1  8.27    NA     14.3 10.6      7 Brent - Ikea BT4  
#>  6 2024-01-01 05:00:00  17.9 12.1     NA     19.5 14.5      6 Brent - Ikea BT4  
#>  7 2024-01-01 06:00:00  18.3 12.5     NA     26   19.3      7 Brent - Ikea BT4  
#>  8 2024-01-01 07:00:00  22.8 16.3     NA     20.8 15.5      8 Brent - Ikea BT4  
#>  9 2024-01-01 08:00:00  24.3 17.3     NA     27.3 20.3     11 Brent - Ikea BT4  
#> 10 2024-01-01 09:00:00  41.8 27.8     NA     31.2 23.2     11 Brent - Ikea BT4  
#> # ℹ 4,630 more rows

Created on 2024-07-15 with reprex v2.1.1

@amystidworthy could you confirm?

Though seconding David that we'd probably direct you to use the more modern approach importing from the locally managed database:

openair::importUKAQ("bt4", year = 2024, source = "local")
#> Warning: ℹ 'local' data is associated with locally managed air quality network sites in
#>   England.
#> ! These sites are not part of the AURN national network, and therefore may not
#>   have the same level of quality control applied to them.
#> This warning is displayed once every 8 hours.
#> # A tibble: 4,662 × 9
#>    source site         code  date                  nox   no2    no    o3  pm10
#>    <chr>  <chr>        <chr> <dttm>              <dbl> <dbl> <dbl> <dbl> <dbl>
#>  1 local  Brent - Ikea BT4   2024-01-01 00:00:00  20.6  12.2   5.5    NA    27
#>  2 local  Brent - Ikea BT4   2024-01-01 01:00:00  21    12.8   5.3    NA    11
#>  3 local  Brent - Ikea BT4   2024-01-01 02:00:00  22.8  14     5.7    NA    11
#>  4 local  Brent - Ikea BT4   2024-01-01 03:00:00  13     8.5   2.9    NA     9
#>  5 local  Brent - Ikea BT4   2024-01-01 04:00:00  12.1   8.3   2.5    NA    11
#>  6 local  Brent - Ikea BT4   2024-01-01 05:00:00  17.9  12.2   3.8    NA    14
#>  7 local  Brent - Ikea BT4   2024-01-01 06:00:00  18.4  12.5   3.8    NA    19
#>  8 local  Brent - Ikea BT4   2024-01-01 07:00:00  22.9  16.3   4.3    NA    15
#>  9 local  Brent - Ikea BT4   2024-01-01 08:00:00  24.3  17.3   4.5    NA    20
#> 10 local  Brent - Ikea BT4   2024-01-01 09:00:00  41.8  27.8   9.1    NA    23
#> # ℹ 4,652 more rows

Created on 2024-07-15 with reprex v2.1.1

amystidworthy commented 4 months ago

Fantastic, thanks Jack and David! I wonder if it was related to the Londonair website SSL certificate having expired, which also seems to be working again now.

From: Jack Davison @.> Sent: 15 July 2024 10:41 To: davidcarslaw/openair @.> Cc: Amy Stidworthy @.>; Mention @.> Subject: Re: [davidcarslaw/openair] Error running importKCL() function to download LondonAir monitoring data (Issue #386)

This appears to be fixed.

openair::importKCL(site="bt4",year=2024)

> Warning in openair::importKCL(site = "bt4", year = 2024): Some of the more

> recent data may not be ratified.

> NOTE - mass units are used

> ug/m3 for NOx, NO2, SO2, O3; mg/m3 for CO

> PM10_raw is raw data multiplied by 1.3

> # A tibble: 4,640 × 9

> date nox no2 o3 pm10_raw pm10 pm25 site code

>

> 1 2024-01-01 00:00:00 20.5 12.1 NA 36.4 27.1 21 Brent - Ikea BT4

> 2 2024-01-01 01:00:00 20.9 12.8 NA 14.3 10.6 10 Brent - Ikea BT4

> 3 2024-01-01 02:00:00 22.7 14.0 NA 14.3 10.6 10 Brent - Ikea BT4

> 4 2024-01-01 03:00:00 13.0 8.51 NA 11.7 8.70 10 Brent - Ikea BT4

> 5 2024-01-01 04:00:00 12.1 8.27 NA 14.3 10.6 7 Brent - Ikea BT4

> 6 2024-01-01 05:00:00 17.9 12.1 NA 19.5 14.5 6 Brent - Ikea BT4

> 7 2024-01-01 06:00:00 18.3 12.5 NA 26 19.3 7 Brent - Ikea BT4

> 8 2024-01-01 07:00:00 22.8 16.3 NA 20.8 15.5 8 Brent - Ikea BT4

> 9 2024-01-01 08:00:00 24.3 17.3 NA 27.3 20.3 11 Brent - Ikea BT4

> 10 2024-01-01 09:00:00 41.8 27.8 NA 31.2 23.2 11 Brent - Ikea BT4

> # ℹ 4,630 more rows

Created on 2024-07-15 with reprex v2.1.1https://reprex.tidyverse.org

@amystidworthyhttps://github.com/amystidworthy could you confirm?

Though seconding David that we'd probably direct you to use the more modern approach importing from the locally managedhttps://uk-air.defra.gov.uk/networks/network-info?view=nondefraaqmon database:

openair::importUKAQ("bt4", year = 2024, source = "local")

> Warning: ℹ 'local' data is associated with locally managed air quality network sites in

> England.

> ! These sites are not part of the AURN national network, and therefore may not

> have the same level of quality control applied to them.

> This warning is displayed once every 8 hours.

> # A tibble: 4,662 × 9

> source site code date nox no2 no o3 pm10

>

> 1 local Brent - Ikea BT4 2024-01-01 00:00:00 20.6 12.2 5.5 NA 27

> 2 local Brent - Ikea BT4 2024-01-01 01:00:00 21 12.8 5.3 NA 11

> 3 local Brent - Ikea BT4 2024-01-01 02:00:00 22.8 14 5.7 NA 11

> 4 local Brent - Ikea BT4 2024-01-01 03:00:00 13 8.5 2.9 NA 9

> 5 local Brent - Ikea BT4 2024-01-01 04:00:00 12.1 8.3 2.5 NA 11

> 6 local Brent - Ikea BT4 2024-01-01 05:00:00 17.9 12.2 3.8 NA 14

> 7 local Brent - Ikea BT4 2024-01-01 06:00:00 18.4 12.5 3.8 NA 19

> 8 local Brent - Ikea BT4 2024-01-01 07:00:00 22.9 16.3 4.3 NA 15

> 9 local Brent - Ikea BT4 2024-01-01 08:00:00 24.3 17.3 4.5 NA 20

> 10 local Brent - Ikea BT4 2024-01-01 09:00:00 41.8 27.8 9.1 NA 23

> # ℹ 4,652 more rows

Created on 2024-07-15 with reprex v2.1.1https://reprex.tidyverse.org

— Reply to this email directly, view it on GitHubhttps://github.com/davidcarslaw/openair/issues/386#issuecomment-2228089451, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ARBCCMPDCR4PYXHGUTIEMITZMOKKBAVCNFSM6AAAAABKVHOTSOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRYGA4DSNBVGE. You are receiving this because you were mentioned.Message ID: @.**@.>>

jack-davison commented 4 months ago

I wonder if it was related to the Londonair website SSL certificate having expired, which also seems to be working again now.

Almost certainly!

Glad it's now working; I'll close this issue.