search

thekangaroofactory / sydney-rain-forecast

ML Rain Forecast in Sydney
0 stars 0 forks source link

App will crash when current observation file is missing #4

Open thekangaroofactory opened 1 year ago

thekangaroofactory commented 1 year ago

Launching the app while the current month observation file is missing will crash the app. Stack: 2023-04-10T18:52:08.746523+00:00 shinyapps[4698443]: Warning in file(file, "rt") : 2023-04-10T18:52:08.746550+00:00 shinyapps[4698443]: cannot open file './dataset/downloads/IDCJDW2124.202304.csv': No such file or directory 2023-04-10T18:52:08.747288+00:00 shinyapps[4698443]: Warning: Error in file: cannot open the connection 2023-04-10T18:52:08.763772+00:00 shinyapps[4698443]: 73: file 2023-04-10T18:52:08.763792+00:00 shinyapps[4698443]: 72: read.table 2023-04-10T18:52:08.763796+00:00 shinyapps[4698443]: 71: read.csv 2023-04-10T18:52:08.763807+00:00 shinyapps[4698443]: 70: getUpdateTime 2023-04-10T18:52:08.763811+00:00 shinyapps[4698443]: 69: module 2023-04-10T18:52:08.763814+00:00 shinyapps[4698443]: 64: callModule 2023-04-10T18:52:08.763833+00:00 shinyapps[4698443]: 63: moduleServer 2023-04-10T18:52:08.763836+00:00 shinyapps[4698443]: 62: weatherDashboard_Server 2023-04-10T18:52:08.763839+00:00 shinyapps[4698443]: 61: server [/srv/connect/apps/SydneyRainForecast/server.R#44] 2023-04-10T18:52:08.763843+00:00 shinyapps[4698443]: 15: 2023-04-10T18:52:08.763849+00:00 shinyapps[4698443]: 13: fn 2023-04-10T18:52:08.763853+00:00 shinyapps[4698443]: 8: retry 2023-04-10T18:52:08.763856+00:00 shinyapps[4698443]: 7: connect$retryingStartServer 2023-04-10T18:52:08.763863+00:00 shinyapps[4698443]: 6: eval 2023-04-10T18:52:08.763866+00:00 shinyapps[4698443]: 5: eval 2023-04-10T18:52:08.763869+00:00 shinyapps[4698443]: 4: eval 2023-04-10T18:52:08.763872+00:00 shinyapps[4698443]: 3: eval 2023-04-10T18:52:08.763878+00:00 shinyapps[4698443]: 2: eval.parent 2023-04-10T18:52:08.763882+00:00 shinyapps[4698443]: 1: local 2023-04-10T18:52:08.763920+00:00 shinyapps[4698443]: Error in file(file, "rt") : cannot open the connection 2023-04-10T18:52:08.827817+00:00 shinyapps[4698443]: Checking missing observations: 2023-04-10T18:52:08.828071+00:00 shinyapps[4698443]: -- latest observation: 19364

It seems like it tries to open the current month file before checking if it's there or not. Then if you relaunch the app, it will work since the file has been downloaded

thekangaroofactory commented 1 year ago

[SERVER] Application server is being started...

Starting reccurentCheck server... Model list loaded, size = 2 Test dataset loaded, size = 1225 24 Warning in file(file, "rt") : cannot open file './dataset/downloads/IDCJDW2124.202305.csv': No such file or directory Warning: Error in file: cannot open the connection 59: file 58: read.table 57: read.csv 56: getUpdateTime [./src/script/R/getUpdateTime.R#9] 55: module [./src/script/R/weather-dashboard/weather_server.R#55] 50: callModule 49: moduleServer 48: weatherDashboard_Server [./src/script/R/weather-dashboard/weather_server.R#16] 47: server [D:\Work\Projects\sydney-rain-forecast\shinyapp/server.R#44] 1: shiny::runApp Error in file(file, "rt") : cannot open the connection Checking missing observations: -- latest observation: 19364 -- isComplete: TRUE Ready to download from http://www.bom.gov.au/climate/dwo/202301/text/IDCJDW2124.202301.csv Download done, size = 4104 Writing file to ./dataset/downloads/IDCJDW2124.202301.csv Ready to download from http://www.bom.gov.au/climate/dwo/202302/text/IDCJDW2124.202302.csv Download done, size = 3824 Writing file to ./dataset/downloads/IDCJDW2124.202302.csv Ready to download from http://www.bom.gov.au/climate/dwo/202303/text/IDCJDW2124.202303.csv Download done, size = 4096 Writing file to ./dataset/downloads/IDCJDW2124.202303.csv Ready to download from http://www.bom.gov.au/climate/dwo/202304/text/IDCJDW2124.202304.csv Download done, size = 4000 Writing file to ./dataset/downloads/IDCJDW2124.202304.csv Ready to download from http://www.bom.gov.au/climate/dwo/202305/text/IDCJDW2124.202305.csv Download done, size = 2128 Writing file to ./dataset/downloads/IDCJDW2124.202305.csv New files to be processed: 5 Processing file: ./dataset/downloads/IDCJDW2124.202301.csv Processing file: ./dataset/downloads/IDCJDW2124.202302.csv Processing file: ./dataset/downloads/IDCJDW2124.202303.csv Processing file: ./dataset/downloads/IDCJDW2124.202304.csv Processing file: ./dataset/downloads/IDCJDW2124.202305.csv Merge observations Processing done. Save formated data

featureEngineering() Load data schema Load mapping Load mean by location Apply feature engineering: -- extract Month from Date -- Fill categorical values out of schema -- Index categorical features -- Fill numerical missing values

  • Dealing with column: MinTemp
  • Dealing with column: MaxTemp
  • Dealing with column: Rainfall
  • Dealing with column: Evaporation
  • Dealing with column: Sunshine
  • Dealing with column: WindGustSpeed
  • Dealing with column: WindSpeed9am
  • Dealing with column: WindSpeed3pm
  • Dealing with column: Humidity9am
  • Dealing with column: Humidity3pm
  • Dealing with column: Pressure9am
  • Dealing with column: Pressure3pm
  • Dealing with column: Cloud9am
  • Dealing with column: Cloud3pm
  • Dealing with column: Temp9am
  • Dealing with column: Temp3pm -- Numerical features normalization Col = MinTemp mean = 12.13181 range= 42.4 Col = MaxTemp mean = 23.17419 range= 52.9 Col = Rainfall mean = 2.326738 range= 371 Col = Evaporation mean = 3.12934 range= 145 Col = Sunshine mean = 3.988338 range= 14.5 Col = WindGustSpeed mean = 37.37759 range= 129 Col = WindSpeed9am mean = 13.86925 range= 130 Col = WindSpeed3pm mean = 18.29286 range= 87 Col = Humidity9am mean = 67.98491 range= 100 Col = Humidity3pm mean = 50.17556 range= 100 Col = Pressure9am mean = 917.3577 range= 60.5 Col = Pressure3pm mean = 915.4338 range= 62.5 Col = Cloud9am mean = 2.762801 range= 9 Col = Cloud3pm mean = 2.695034 range= 9 Col = Temp9am mean = 16.87951 range= 47.4 Col = Temp3pm mean = 21.27147 range= 52.1 << featureEngineering() Save processed data