Closed bhaules closed 2 years ago
bhaules
commented
2 years ago 
I just went in my google api account and clicked around and found that I was going over my quota every day and probably my quota was reset at in the evening and fill it up by morning. Still don't know why I'm going over my quota in just a few hours.
bhaules
commented
2 years ago Anyone having any ideas?
eouia
commented
2 years ago You might have too many folders and photos in your "FamilyPhotos" and too often scanning. Your config shows, it tries to scan your album every 10 minutes. At each scan, it will consume quite a lot of quota due to your amount of photos.
My suggestions;
eouia
commented
2 years ago PS; Over 1-hour scanInterval might not be a good idea. As far as I remember, Google Photos' temporal instant URL of photos would be expired after 1 hour from request. Anyway, your photos are too many to handle with a limited quota.
bhaules
commented
2 years ago thanks mate for your help.
I'll try scanning less often, didn't realise that this was killing my quota (to be honest I didn't really understand what it did until now).
My config has been the same all this time and the photos in the album grew by a few a day.
This bit here makes me think I have 2 instances running in the same time, or maybe the scanning hasn't finished in the 10 minutes and it started a new instance of scanning the photos?
263684 [19.12.2021 02:02.36.335] [LOG] [GPHOTOS:CORE] Indexing photos now. total: 5240
263685 [19.12.2021 02:02.39.673] [LOG] [GPHOTOS:CORE] Indexing photos now. total: 5289
263686 [19.12.2021 02:02.39.770] [LOG] [GPHOTOS:CORE] Indexing photos now. total: 651
263687 [19.12.2021 02:02.41.940] [LOG] [GPHOTOS:CORE] Indexing photos now. total: 5338
263688 [19.12.2021 02:02.42.876] [LOG] [GPHOTOS:CORE] Indexing photos now. total: 696
263689 [19.12.2021 02:02.45.429] [LOG] [GPHOTOS:CORE] Indexing photos now. total: 5388
263690 [19.12.2021 02:02.46.248] [LOG] [GPHOTOS:CORE] Indexing photos now. total: 734
263691 [19.12.2021 02:02.48.599] [LOG] [GPHOTOS:CORE] Indexing photos now. total: 778
263692 [19.12.2021 02:02.49.679] [LOG] [GPHOTOS:CORE] Indexing photos now. total: 5438Your opinion is Interesting, it could be possible.
Hmmm.. Is MM or module executed twice? Probably you might check that thing already...
I believe the module owner @aneaville could help you. This module have been modified since the ownership was transferred from me to @aneaville, So my knowledge about this module is somehow out-of-date. I hope he can diagnose deeper and help you.
bhaules
commented
2 years ago pi@MagicMirror2:~/MagicMirror/modules $ pm2 restart all
Use --update-env to update environment variables
[PM2] Applying action restartProcessId on app [all](ids: [ 0 ])
[PM2] [MagicMirror](0) ✓
┌─────┬────────────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id │ name │ namespace │ version │ mode │ pid │ uptime │ ↺ │ status │ cpu │ mem │ user │ watching │
├─────┼────────────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 0 │ MagicMirror │ default │ 2.17.1 │ fork │ 1324 │ 0s │ 8 │ online │ 0% │ 2.7mb │ pi │ disabled │
└─────┴────────────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘I don't think I'm running the module twice, but I don't know how to find that out other than the above
eouia
commented
2 years ago If you run MM twice by mistake, it will spit out error earlier because the same port is used. So this is not the case. I suspect the algorithm of module, using setInterval instead of my old setTimeout method, @aneaville did. Usually that approach will work well, it will prevent the stop of module while not severe error happens. My old code was stopping module on meeting even tiny problem. So probably his modification is advanced. But in this case, I suspect module is trying to scan forcely despite of some issue, so the connections might be established twice.
I’m not sure really what happened. I hope @aneaville come back and solve our issues.
aneaville
commented
2 years ago From the times in your log, it took about 1.5 min to scan from 4230 to 5488. 63628 [19.12.2021 02:01.29.956] [LOG] [GPHOTOS:CORE] Indexing photos now. total: 4230 263694 [19.12.2021 02:02.52.221] [LOG] [GPHOTOS:CORE] Indexing photos now. total: 5488
You have 6288 photos in the album to scan, so I expect a full scan to take about 7.5 min. This can get slowed down by many things though, and it will start a new scan in parallel if the scan interval is too fast. I'm not sure if things work nicely in this case or if counters will get shared and mess things up.
But I do expect slowing the scan interval down will solve the issue. Scanning every 10 min ate up your quota and you are correct that your quota gets reset every 24 hours. I recommend increasing the scan interval to 60 min or longer.
The scan just gets a list of all the photos the module needs to show. This list rarely changes unless you add a new photo to the album. Detecting within an hour is usually fine. Also since the scan takes so long, the module saves that photo list as a file(just the IDs, not the actual photos), so it can start showing pictures faster. So once a scan is completed successfully, the module could work even if the scan fails continuously.
Eouia is right though, the link for a photo is only good for about an hour. Another change I made was to 'refresh' the link to the photos that are getting showed. So scanning at slower than 60 min is ok. The refreshing uses the same quota though, and if it can't refresh the links, it will stop working.
Hope this helps, sorry for being slow to respond. Please let us know if changing the scan interval to 60 min solves your issues.
bhaules
commented
2 years ago But I do expect slowing the scan interval down will solve the issue. Scanning every 10 min ate up your quota and you are correct that your quota gets reset every 24 hours. I recommend increasing the scan interval to 60 min or longer. I guess where this seems a bit odd is the fact that I had this configuration for a full year and it only started doing this after I re-installed Magic Mirror and all the modules. So in theory can I set the scan to happen once a day? In this way it will scan my 6000 photos only once. How many requests will it eat when scanning 6000 photos? Before the reinstallation this is how my quota looked:
Steps I will take: Set the scan interval to 24 hours assuming that the links are being refreshed. If it doesn't work to 60 mins Would a full re-install of the module do anything? How many requests do I make to the Google API for each scan?
Thank you for taking your time to respond to this. You don't
aneaville
commented
2 years ago I love the annotated graph.
The most likely cause is the change that made the code more robust by using an interval timer rather than waiting for the scan to complete before starting a timer. Old way: start scan - finish scan - wait time - start scan New way: Every 'wait time' kick off new scan regardless if previous one is done. This would make the scan happen more frequently with the same settings.
A contributing factor is that the list is 'refreshed' before using, which uses additional API calls.
With these 2 factors, and the settings you had, I suspect your API calls would increase by 3x, which explains the graphs you show.
Google limits API requests to only get 50 photos at a time, so 6000 photos will take 6000/50 = 120 API calls. With a 10 minute scan interval, 24 hours 60 min / 10 min = 144 scans per day. Thats 144 scans120 API calls = 17,280 API calls just for scans. Every 20 min the links are 'refreshed' with a single API call, using an extra 24 hours * 60 min / 20min per refresh = 72 calls.
With the new changes, fast scanning no longer helps anything other than detecting newly added photos faster. With 6000 photos and randomly showing, its doubtful that new photos would be noticed that soon.
Hope this helps explain things a bit.
I don't expect a full reinstall would change anything. Scan interval of 24 hours should work, but 1 or 2 hours should be fine.
bhaules
commented
2 years ago Thank you very much for your detailed response.
Changed the scan time to 30 minutes and it solved my issue.
My API calls are down to 3500. Thanks for making me understand how things work under the hood.
Thanks again. You can clos this ticket
I had this configuration for more than a year and recently re-installed Magic Mirror and all the modules. This is my GooglePhotos config:
When I go to the MagicMirror-out.log I get:
When I go to MagicMirror-error.log I get:
What I found about this error was this page from the forum: https://forum.magicmirror.builders/topic/14871/mmm-googlephotos-unhandled-promise-rejection/8 but I can't seem to figure out how am I getting too may requests as I should be below that number with a request every 5 minutes.
In the evenings though the Google Photos module seems to start working. Last night it worked from about 8PM to about 2AM:
Looking at the logs what I found curious was that it seems that there are multiple streams of indexing happeing in paralel:
Does anybody have an idea on what I should try to fix this? Thank you