search

sdetweil / MyCovid19

monitor COV ID 19 status
9 stars 5 forks source link

no more graphs showing up - data changed from daily to weekly? #13

Closed waldmeisterx closed 3 years ago

waldmeisterx commented 3 years ago

Since some days, I don't get any graphs shown. When I compare the size of the current countries-rawdata-*.csv file with the size of older ones it is a lot smaller. Maybe they changed the content to weekly date instead of daily? (same for xml and json) The only file containing daily data is xlsx.

waldmeisterx commented 3 years ago

I found this here https://www.ecdc.europa.eu/en/publications-data/download-todays-data-geographic-distribution-covid-19-cases-worldwide

"ECDC switched to a weekly reporting schedule for the COVID-19 situation worldwide and in the EU/EEA and the UK on 17 December this year. Hence, all daily updates have been discontinued from 14 December. ECDC will publish updates on the number of cases and deaths reported worldwide and aggregated by week every Thursday. The weekly data will be available as downloadable files in the following formats: XLSX, CSV, JSON and XML. As an exception, the weekly updates for the end-of-year festive season will be published on 23 December and 30 December 2020."

sdetweil commented 3 years ago

well isn't that special?!

thanks for the info. will have to look at the impact

sdetweil commented 3 years ago

I have uploaded changes to support a different source... still daily.. please review

do a git pull in the module folder to get the updates

waldmeisterx commented 3 years ago

Thanks for the quick reply, so here a quick first response. I updated the module (git pull, npm install) and restarted mm. The new data file (now 13MB) is downloaded, but no graph is displayed. Always telling "loading data" and mm is restarting frequently. Further analysis on my side can be done tomorrow...

sdetweil commented 3 years ago

ok, add debug:true to the module config.

what country names are you using

waldmeisterx commented 3 years ago

I have configured Germany and Austria: countries: ['Germany','Austria'],

sdetweil commented 3 years ago

Screenshot at 2020-12-19 07-05-32

seems ok here 

 {
            module: "MyCovid19",
            disabled: false,
            position: "top_center",
            classes: "sam",
            config: {
                countries: [
                    "Germany",
                    "Austria",
                ],
                // line colors can be any definition of color either a name ,or a hex string
                // one per country above, used in order,
                line_colors: [
                    "red",
                    "white",
                ],
                //
                chart_type: "cases", // or "cumulative deaths"
                chart_title: "daily new cases ", // however u want to label
                // the vertical steps on the chart.. how tall u want it to be and how mant increments
                ranges: { min: 0, max: 4000, stepSize: 500 },
                // size of the chart in pixels
                width: 400,
                height: 400,
                // only used if we need to debug something
                //debug:true,
                startDate: "03/13/2020",
                debug: true,
            },
        },
waldmeisterx commented 3 years ago

OK, now got this one running too! I updated my config a little (changed from some ' to ", added some spaces in the ranges and removed more configured lines than countries). I run two graphs like the following config. But I can run only one at a time, if I enable both, mm starts restarting again. ` { module: "MyCovid19", disabled: false, position: "bottom_left", config: { countries: [ "Germany", "Austria", ], // one per country above, used in order, line_colors: [ "red", "green", ], chart_type: "cases", chart_title: "Fälle/Tag", startDate: "03/01/2020", // the vertical steps on the chart.. how tall u want it to be and how many increments ranges:{ min: 0, max: 40000, stepSize: 10000}, xAxisLabel: "Datum", yAxisLabel: "Infektionen", // size of the chart in pixels width: 500, height: 350, // only used if we need to debug something debug: false, }, },

    {
        module: "MyCovid19",
        disabled: false,
        position: "bottom_right",
        config: {
            countries: [
                "Germany",
                "Austria",
            ],
            // one per country above, used in order,
            line_colors: [
                "red",
                "green",
            ],
            chart_type: "cumulative_cases",
            chart_title: "Kumulierte Fälle",
            startDate: "03/01/2020",
            // the vertical steps on the chart.. how tall u want it to be and how many increments
            ranges:{ min: 0, max: 1500000, stepSize: 250000},
            xAxisLabel: "Datum",
            yAxisLabel: "Infektionen",
            // size of the chart in pixels
            width: 500,
            height: 350,
            // only used if we need to debug something
            debug: false,
        },
    },

` And I can't get any helpful debug output.

sdetweil commented 3 years ago

for debug, have to change debug to true i have 10 different charts at once, used your config, copy/paste, no changes, and it works ok can u make a config.js with only my module enabled?

Screenshot at 2020-12-19 09-00-00

waldmeisterx commented 3 years ago

Yes, I know for debug output to set the parameter to true. I did that. And I can't get only two of your modules to work. :-(

Here my complete config (I only shortened the ipWhitelist) `var config = { // address: "localhost", // Address to listen on, can be: address: "0.0.0.0", port: 8080, basePath: "/", ipWhitelist: ["127.0.0.1", "::ffff:127.0.0.1", "::1"], useHttps: false, // Support HTTPS or not, default "false" will use HTTP httpsPrivateKey: "", // HTTPS private key path, only require when useHttps is true httpsCertificate: "", // HTTPS Certificate path, only require when useHttps is true language: "de", logLevel: ["WARN", "ERROR"], //["INFO", "LOG", "WARN", "ERROR"], timeFormat: 24, units: "metric",

modules: [

/**

/**

/ DO NOT EDIT THE LINE BELOW / if (typeof module !== "undefined") {module.exports = config;}`

and this is the debug output `

magicmirror@2.13.0 start /home/pi/MagicMirror DISPLAY="${DISPLAY:=:0}" ./node_modules/.bin/electron js/electron.js

[2020-12-19 17:13:58.360] [LOG] Starting MagicMirror: v2.13.0 [2020-12-19 17:13:58.373] [LOG] Loading config ... [2020-12-19 17:13:59.824] [LOG] Starting node_helper for module: MyCovid19 [2020-12-19 17:14:03.362] [LOG] requested file exists=modules/MyCovid19/countries-rawdata-12-19-2020.csv sending back to module_0_MyCovid19_7621 [2020-12-19 17:14:03.363] [LOG] ready to get data id=module_0_MyCovid19_7621 [2020-12-19 17:14:03.367] [LOG] check for file =modules/MyCovid19/countries-rawdata-12-19-2020.csv [2020-12-19 17:14:03.372] [LOG] processing id=module_0_MyCovid19_7621 file=modules/MyCovid19/countries-rawdata-12-19-2020.csv fields=[object Object] [2020-12-19 17:14:03.429] [LOG] requested file exists=modules/MyCovid19/countries-rawdata-12-19-2020.csv sending back to module_1_MyCovid19_1941 [2020-12-19 17:14:03.430] [LOG] ready to get data id=module_1_MyCovid19_1941 [2020-12-19 17:14:03.432] [LOG] check for file =modules/MyCovid19/countries-rawdata-12-19-2020.csv [2020-12-19 17:14:03.433] [LOG] processing id=module_1_MyCovid19_1941 file=modules/MyCovid19/countries-rawdata-12-19-2020.csv fields=[object Object]

magicmirror@2.13.0 start /home/pi/MagicMirror DISPLAY="${DISPLAY:=:0}" ./node_modules/.bin/electron js/electron.js

[2020-12-19 17:15:34.256] [LOG] Starting MagicMirror: v2.13.0 [2020-12-19 17:15:34.268] [LOG] Loading config ... [2020-12-19 17:15:35.709] [LOG] Starting node_helper for module: MyCovid19 [2020-12-19 17:15:39.077] [LOG] requested file exists=modules/MyCovid19/countries-rawdata-12-19-2020.csv sending back to module_0_MyCovid19_7911 [2020-12-19 17:15:39.078] [LOG] ready to get data id=module_0_MyCovid19_7911 [2020-12-19 17:15:39.082] [LOG] check for file =modules/MyCovid19/countries-rawdata-12-19-2020.csv [2020-12-19 17:15:39.084] [LOG] processing id=module_0_MyCovid19_7911 file=modules/MyCovid19/countries-rawdata-12-19-2020.csv fields=[object Object] [2020-12-19 17:15:39.101] [LOG] requested file exists=modules/MyCovid19/countries-rawdata-12-19-2020.csv sending back to module_1_MyCovid19_6161 [2020-12-19 17:15:39.104] [LOG] ready to get data id=module_1_MyCovid19_6161 [2020-12-19 17:15:39.108] [LOG] check for file =modules/MyCovid19/countries-rawdata-12-19-2020.csv [2020-12-19 17:15:39.109] [LOG] processing id=module_1_MyCovid19_6161 file=modules/MyCovid19/countries-rawdata-12-19-2020.csv fields=[object Object] [2020-12-19 17:15:53.257] [LOG] requested file exists=modules/MyCovid19/countries-rawdata-12-19-2020.csv sending back to module_0_MyCovid19_8951 [2020-12-19 17:15:53.262] [LOG] ready to get data id=module_0_MyCovid19_8951 [2020-12-19 17:15:53.265] [LOG] check for file =modules/MyCovid19/countries-rawdata-12-19-2020.csv [2020-12-19 17:15:53.266] [LOG] processing id=module_0_MyCovid19_8951 file=modules/MyCovid19/countries-rawdata-12-19-2020.csv fields=[object Object] [2020-12-19 17:15:53.281] [LOG] requested file exists=modules/MyCovid19/countries-rawdata-12-19-2020.csv sending back to module_1_MyCovid19_3811 [2020-12-19 17:15:53.282] [LOG] ready to get data id=module_1_MyCovid19_3811 [2020-12-19 17:15:53.284] [LOG] check for file =modules/MyCovid19/countries-rawdata-12-19-2020.csv [2020-12-19 17:15:53.285] [LOG] processing id=module_1_MyCovid19_3811 file=modules/MyCovid19/countries-rawdata-12-19-2020.csv fields=[object Object]

magicmirror@2.13.0 start /home/pi/MagicMirror DISPLAY="${DISPLAY:=:0}" ./node_modules/.bin/electron js/electron.js

[2020-12-19 17:17:17.378] [LOG] Starting MagicMirror: v2.13.0 [2020-12-19 17:17:17.392] [LOG] Loading config ... [2020-12-19 17:17:18.881] [LOG] Starting node_helper for module: MyCovid19 [2020-12-19 17:17:22.933] [LOG] requested file exists=modules/MyCovid19/countries-rawdata-12-19-2020.csv sending back to module_0_MyCovid19_3211 [2020-12-19 17:17:22.934] [LOG] ready to get data id=module_0_MyCovid19_3211 [2020-12-19 17:17:22.938] [LOG] check for file =modules/MyCovid19/countries-rawdata-12-19-2020.csv [2020-12-19 17:17:22.940] [LOG] processing id=module_0_MyCovid19_3211 file=modules/MyCovid19/countries-rawdata-12-19-2020.csv fields=[object Object] [2020-12-19 17:17:22.960] [LOG] requested file exists=modules/MyCovid19/countries-rawdata-12-19-2020.csv sending back to module_1_MyCovid19_5161 [2020-12-19 17:17:22.961] [LOG] ready to get data id=module_1_MyCovid19_5161 [2020-12-19 17:17:22.963] [LOG] check for file =modules/MyCovid19/countries-rawdata-12-19-2020.csv [2020-12-19 17:17:22.964] [LOG] processing id=module_1_MyCovid19_5161 file=modules/MyCovid19/countries-rawdata-12-19-2020.csv fields=[object Object]

magicmirror@2.13.0 start /home/pi/MagicMirror DISPLAY="${DISPLAY:=:0}" ./node_modules/.bin/electron js/electron.js

[2020-12-19 17:18:47.844] [LOG] Starting MagicMirror: v2.13.0 [2020-12-19 17:18:47.858] [LOG] Loading config ... [2020-12-19 17:18:49.296] [LOG] Starting node_helper for module: MyCovid19 [2020-12-19 17:18:52.861] [LOG] requested file exists=modules/MyCovid19/countries-rawdata-12-19-2020.csv sending back to module_0_MyCovid19_951 [2020-12-19 17:18:52.862] [LOG] ready to get data id=module_0_MyCovid19_951 [2020-12-19 17:18:52.864] [LOG] check for file =modules/MyCovid19/countries-rawdata-12-19-2020.csv [2020-12-19 17:18:52.866] [LOG] processing id=module_0_MyCovid19_951 file=modules/MyCovid19/countries-rawdata-12-19-2020.csv fields=[object Object] [2020-12-19 17:18:52.918] [LOG] requested file exists=modules/MyCovid19/countries-rawdata-12-19-2020.csv sending back to module_1_MyCovid19_8401 [2020-12-19 17:18:52.920] [LOG] ready to get data id=module_1_MyCovid19_8401 [2020-12-19 17:18:52.922] [LOG] check for file =modules/MyCovid19/countries-rawdata-12-19-2020.csv [2020-12-19 17:18:52.923] [LOG] processing id=module_1_MyCovid19_8401 file=modules/MyCovid19/countries-rawdata-12-19-2020.csv fields=[object Object] `

sdetweil commented 3 years ago

thx. will review

sdetweil commented 3 years ago

i don't see any issues at all.. using your config, changed port Screenshot at 2020-12-19 11-32-47

mm restarting has got to be some other problem..

sdetweil commented 3 years ago

i would reinstall MM, and try that rename the existing folder out of the way (rename to some other name)

waldmeisterx commented 3 years ago

ok, I'll try that...

waldmeisterx commented 3 years ago

is there any other (debug)output from mm that could be analysed to help identifying the problem?

sdetweil commented 3 years ago

not that I am aware of.. each module may or may not have debug.. most of the default modules do not have very much

waldmeisterx commented 3 years ago

Now I installed a new MM after renaming old folder. Only added module MyCovid19. Used old config from above. Started MM this way (not via pm2 restart mm) and got this JS stacktrace

` pi@mirror:~/MagicMirror $ DISPLAY=:0 npm start

magicmirror@2.13.0 start /home/pi/MagicMirror DISPLAY="${DISPLAY:=:0}" ./node_modules/.bin/electron js/electron.js

[2020-12-19 20:38:10.131] [LOG] Starting MagicMirror: v2.13.0 [2020-12-19 20:38:10.149] [LOG] Loading config ... [2020-12-19 20:38:11.794] [LOG] Starting node_helper for module: MyCovid19 [2020-12-19 20:38:15.414] [LOG] requested file exists=modules/MyCovid19/countries-rawdata-12-19-2020.csv sending back to module_0_MyCovid19_8361 [2020-12-19 20:38:15.416] [LOG] ready to get data id=module_0_MyCovid19_8361 [2020-12-19 20:38:15.421] [LOG] check for file =modules/MyCovid19/countries-rawdata-12-19-2020.csv [2020-12-19 20:38:15.424] [LOG] processing id=module_0_MyCovid19_8361 file=modules/MyCovid19/countries-rawdata-12-19-2020.csv fields=[object Object] [2020-12-19 20:38:15.448] [LOG] requested file exists=modules/MyCovid19/countries-rawdata-12-19-2020.csv sending back to module_1_MyCovid19_4261 [2020-12-19 20:38:15.452] [LOG] ready to get data id=module_1_MyCovid19_4261 [2020-12-19 20:38:15.457] [LOG] check for file =modules/MyCovid19/countries-rawdata-12-19-2020.csv [2020-12-19 20:38:15.458] [LOG] processing id=module_1_MyCovid19_4261 file=modules/MyCovid19/countries-rawdata-12-19-2020.csv fields=[object Object]

<--- Last few GCs ---> t of marking 1141 ms) (average mu = 0,146, current mu = 0,020) allocation fai[2865:0x67e8000] 31289 ms: Mark-sweep 217,7 (218,9) -> 217,7 (219,9) MB, 965,3 / 0,0 ms (+ 50,0 ms in 10 steps since start of marking, biggest step 7,0 ms, walltime since start of marking 1037 ms) (average mu = 0,087, current mu = 0,021) allocation fa[2865:0x67e8000] 32319 ms: Mark-sweep 218,5 (219,9) -> 217,8 (219,9) MB, 1012,3 / 0,0 ms (average mu = 0,053, current mu = 0,017) allocation failure scavenge might not succeed

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x242d0b21 0: builtin exit frame: concat(this=0x593ded11 <JSArray[44274]>,0x485fd09d <JSArray[226]>,0x593ded11 <JSArray[44274]>)

1: appendFinalResult [0x36e6fe15] [/home/pi/MagicMirror/modules/MyCovid19/node_modules/csvtojson/v2/Result.js:68] [bytecode=0x384f3509 offset=19](this=0x52e49629 <Result map = 0x20dc577d>,0x485fd09d <JSArray[226]>)
2: /* anonymous */(aka /* anonymous */) [0x27650185] [/h...

` is this helpful?

sdetweil commented 3 years ago

what device is this on? looks like u are running out of memory (which would be fatal).. the file has gotten bigger

u may need to increase the swap space

https://pimylifeup.com/raspberry-pi-swap-file/

waldmeisterx commented 3 years ago

It's a raspberry 3 and swapsize was configured to 100, changed it to 1024, rebooted. The problem is still there:

`[2020-12-19 21:27:58.242] [LOG] processing id=module_1_MyCovid19_9541 file=modules/MyCovid19/countries-rawdata-12-19-2020.csv fields=[object Object]

<--- Last few GCs --->

[1890:0x66ee000] 32325 ms: Mark-sweep 217,4 (219,1) -> 217,2 (218,9) MB, 1657,9 / 0,2 ms (+ 0,0 ms in 1 steps since start of marking, biggest step 0,0 ms, walltime since start of marking 1668 ms) (average mu = 0,075, current mu = 0,010) finalize incre

<--- JS stacktrace --->

==== JS stack trace =========================================

0: ExitFrame [pc: 0x1efe4c4]
1: StubFrame [pc: 0x1e72784]

Security context: 0x6d510b21 2: processTicksAndRejections [0x31301315] [internal/process/task_queues.js:90] [bytecode=0x5e75acb1 offset=164](this=0x31301199 ) 3: InternalFrame [pc: 0x1e6f230] 4: EntryFrame [pc: 0x1e6ef34]

==== Details ================================================

[0]: ExitFrame [pc: 0x1efe4c4]

`

sdetweil commented 3 years ago

what are the versions of node and npm

node --version npm --version also free -m

waldmeisterx commented 3 years ago 
node --version v10.23.0
npm --version  6.14.8
free -m
              total        used        free      shared  buff/cache   available
Mem:            873         117         420           1         335         702
Swap:          1023          14        1009
waldmeisterx commented 3 years ago

with running MM (with one MyCovid19 module):

free -m
              total        used        free      shared  buff/cache   available
Mem:            873         220         288          13         364         587
Swap:          1023          14        1009
sdetweil commented 3 years ago

lets check the file download, in the MyCovid19 folder do md5sum countries-rawdata-12-19-2020.csv

i got 63ad10ea8c231d7270fab646f8e6ca68

waldmeisterx commented 3 years ago

63ad10ea8c231d7270fab646f8e6ca68 countries-rawdata-12-19-2020.csv ok, the same here

sdetweil commented 3 years ago 
[2020-12-19 12:32:30.781] [LOG]    processing id=module_17_MyCovid19_321 file=modules/MyCovid19/states-rawdata-12-19-2020.csv fields=[object Object]
[2020-12-19 12:32:31.555] [LOG]    done processing file id=module_17_MyCovid19_321

so those two messages are before and after we run the file thru the csv to json converter library (we are at the latest level on that)

to reduce memory and time, as cvttojson reads the file line by line, i check to see if the location on this record matches any specified in the config. if true, then i check the date matches on or after the start date if both are true, i tell cvttojson to save the line

the last message comes out after cvtto json has processed all the lines and is handing me a workable javascript object.

we are crashing in there somewhere. cvttojson doesn't have any reported errors like this, or memory related.

how old is your sd card? sadly they are not all they are cracked up to be..
they have a fixed number of bits for storing data, and these bits can only be used ONCE.. once written to, the NEXT time there is a write, that prior bit is marked used, and a new one must be allocated by the hardware.. when u run out, you run out, but start getting weird errors... nothing reports card consumed...

this works great for low write environments like cameras and music players.. but not so good for operating systems that are constantly writing..

i'm just thinking out loud here...

waldmeisterx commented 3 years ago

I do not really know, how old this sd card is. It is maybe possible, that it has problems. I'll give it a try, but have to present my a new one for xmas. This may take some days, but I will come back with the result...

waldmeisterx commented 3 years ago

I found a brand new SanDisk Ultra 32GB card and finally did a complete fresh install of all, but the error is still there. I did a complete doku of all steps I did for setting the pi up, which I'll send to yo by mail. The new JS dumps are included. Maybe it depends on other modules I use?? And... no hurry, it's a few days before xmas and other things may be more urgent...

sdetweil commented 3 years ago

well that is something..
same user ID at Gmail. very interested.

but it failed w just my module too, right?

u reinstalled everything, not restored some backup, right?

waldmeisterx commented 3 years ago

Yes, I did a complete new install, NO restore (only config.js and custom.css are restored). And yes it fails with a config with only twice your module, too.

(now I have to go to bed, it's 3:30 am here, more tomorrow...)

sdetweil commented 3 years ago

good news, i can recreate it on my pi 3. more news to follow when i have it

waldmeisterx commented 3 years ago

these are great news to hear it is not special to my hw or config :-)

sdetweil commented 3 years ago

I opened an issue on the cvttojson module https://github.com/Keyang/node-csvtojson/issues/414

waldmeisterx commented 3 years ago

ok, great, let's wait and see (btw. I think file site is 13M not 13 gig!?)

sdetweil commented 3 years ago

corrected the file size..

sdetweil commented 3 years ago

@waldmeisterx in the module folder do 

git pull

then 

git checkout newthread

and give it a try..

waldmeisterx commented 3 years ago

This looks good! :-) I can see both of my configured graphs now! MM is not crashing.

sdetweil commented 3 years ago

lets run with this for a while, I have pi3 with your charts and pi4 with 10 charts, countries, states, and US counties.

waldmeisterx commented 3 years ago

ok, I'll keep you updated nice work

sdetweil commented 3 years ago

i think there is a bug in daily update.. will look at it tomorrow.. got logging on..

sdetweil commented 3 years ago

@waldmeisterx do another git pull in the module folder, I think I have resolved it.. the old file the last date was the same as today, the new file the last date is yesterday... so the dates never matched, and there was a bug in the recheck logic..

waldmeisterx commented 3 years ago

@sdetweil Sorry for the delay, but now I did a git pull and updated my MyCovid19 module. After MM restart it looks good at the moment.

waldmeisterx commented 3 years ago

@sdetweil some status updates: After updating the module after your last info it only once updated the data over night. I disabled my workaround by restarting the mm with cron in the morning. The current data file is still countries-rawdata-12-28-2020.csv So i reanable the restart again.

sdetweil commented 3 years ago

hm... mine continues to update..

can u turn on debug in the modules config debug: true

and start MM with npm start >somefile (don't use pm2 make logs hard to use)

and let it run. at least 2 days (2 midnights), thanks...

waldmeisterx commented 3 years ago

ok, stopped mm via pm2, debug: true is still set, restarted mm via npm debug file is growing... had to disable module MMM-MQTT because it throws a lot debug output to debug file, but has no switch to disable debug output :(

sdetweil commented 3 years ago

is this still failing?

waldmeisterx commented 3 years ago

@sdetweil no, it was running until 22.4.2021

Then i got this error MM crashing (after pi os update?) where we had contact. I disabled the module since i get these new errors.