Google Drive upload - creates directories, but no files uploaded

[jetmonk]

motionEye Version 0.42.1; Motion Version 4.0, OS Version Raspbian 9.4

Hello,

I recently updated motioneye to 0.42.1, and I'm having problems with Google Drive. I can get a token, but when running 'Test Service' with the token it usually times out, and rarely gives a success message. Sometimes it says to reload the page; usually the wheel spins and goes away without a message. But it does create the directory of my choosing on Google Drive, so auth works. It seems to do this with and without backslashes in the Drive pathname.

However, it never uploads a photo. I test this by clicking on the manual photo icon. A photo shows up in local storage, but nothing ever makes it to Drive.

I haven't been able to find this specific issue - creating directory but no uploads - in the issues list.

edit: Finally, some images showed up, but in wrong directory for the camera in question (perhaps propagation delays from past setting?). And manual snapshots are no longer working, not even for local memory, for the 3rd party network camera. Specifically, I have an Arecont drive directory for the net cam, and an RPi-3 directory for the native Pi camera, and the Pi photos are ending up in the Arecont directory.

These problems are so all over the place that I don't think this constitutes an actionable bug report. Other people have successfully tried to fix Drive problems with an SD card format. So I'm going to close it rather than give maintainers a hairball of a bug report, and study the problem myself.

[jetmonk]

Closing issue because there are too many issues to be an actionable bug report.

[jetmonk]

Closing issue because there are too many issues to be an actionable bug report. I need to experiment with reinstallation, recreating all configs, etc.

[MichaIng]

There have been some fixes for media uploads. Could you try motionEye v0.43.0 from the dev branch? https://github.com/motioneye-project/motioneye/tree/dev#installation

[jetmonk]

I upgraded my RPi to Buster (to get python 3.7, needed for the dev version).

Now

1. I could connect to google drive

2. it seemed to upload some files to Drive but ...

3. … the wrong camera’s images are in the Drive directory (contrary to uploadservices.json) - so RPi3 camera is putting files into Arecont’s Drive directory, while the Arecont isn’t putting images anywhere, though they were both briefly putting files into the right directory before I deleted the Drive config to start anew.

4. There seems to be an unwanted intermediate subdirectory /nt/ in the Drive paths (maybe a fragment of ‘Arecont’, a name of a camera)

5. Motion triggered images don’t seem to work even though red boxes seem to show up when motion is detected.

6. The manual camera-icon photos never work. No snapshots are created.

Next, I I turned off Drive sharing. I deleted the toplevel Security google drive directory (of Security/MotionEye/nt//. I stopped Motioneye with 'sudo service motioneye stop'. I deleted uploadservices.json. I restarted motioneye. I totally reconfigured Drive with new tokens giving above directory paths by camera name, and got new keys. This rebuilt paths Security/MotionEye//.

Nothing is showing up on Google Drive except for the empty paths, despite interval photos being taken. I tried doing a stop/start of the service. Still nothing.

The test of the upload says "Accessing the upload service succeeded![object Object]” which seems to be a slight programming glitch.

Then I tried deleting the files on Drive, turning uploads off then on. Drive directories were not re-creatred. Then I hit ’Test service’ and empty directories were recreated for Arecont camera, but RPi3 camera lost its info in uploadservices.json. After getting another key, the Rpi3 directory was recreated, but now no file uploads are occurring despite an interval image being made very minute.

So … I basically can’t get anything to work. In particular, I’m confused that the manual photo button never works. And motion triggers don’t work, though I see evidence of motion.

On Jun 3, 2022, at 3:02 PM, MichaIng @.***> wrote:

There have been some fixes for media uploads. Could you try motionEye v0.43.0 from the dev branch? https://github.com/motioneye-project/motioneye/tree/dev#installation https://github.com/motioneye-project/motioneye/tree/dev#installation — Reply to this email directly, view it on GitHub https://github.com/motioneye-project/motioneye/issues/2516#issuecomment-1146273886, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALIXYESTRO7RQP4C6SYYPOTVNJJE7ANCNFSM5XZF2LJA. You are receiving this because you modified the open/close state.

[MichaIng]

Can you please show the logs:

journalctl -u motioneye

And can you show the two camera configs (mask Google credentials, of course):

cat /etc/motioneye/camera*.conf

[jetmonk]

Thank you for having a look - please see https://github.com/jetmonk/motioneye-issues-2022-07 for my anonymized config files, screenshots, and a NOTES.txt file.

I wiped my /etc/motioneye directory and reinstalled cameras. I was mostly successful but I think that there were some glitches along the way. This is described in NOTES-RECONFIG-FROM-SCRATCH.txt - I guess I get the feeling that there was some order-based weirdness as described in this file - eg, 2nd camera behaved a bit differently - and I couldn't remotely configure Google Drive from an Rpi2 slave camera (running old motioneye). Snapshots now work, when before they didn't.

edit: now it seems that Drive uploads don't work with motion detection (motion detected images show up locally); but reverting to manual images restores drive uploads.

[BettySwallocks]

I'm interested to know how this is going as I have some similar issues to the OP. My Google Drive uploads had been working fine, but stopped after I made a couple of basic changes. I incorrectly set an 'every frame' upload while testing (resulting in something like 3000 images), but corrected that back to only 1 image on motion. I cleared the unwanted Google Drive files the easy way by clearing a whole date folder under my camera name. After that I can't get any new images on Drive, regardless of resetting the access/key or changing options away from Google Drive and back again. My situation may be slightly complicated by the fact that I'm running Home Assistant Supervised via a Docker container in my RasPi3 and then have added MotionEye as an Add-On component. Was working great for a short while...

[BettySwallocks]

Further - I've tried a number of the actions suggested both by and to the OP, and again have similar findings. My motioneye logs have these entries each time I take a new snapshot for upload:

ERROR: gdrive: failed to obtain credentials: Bad Request ERROR: failed to upload file "/share/motioneye/Camera1/2022-06-20/11-44-33.jpg" with service gdrive: Bad Request Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/motioneye/uploadservices.py", line 1103, in upload_media_file service.upload_file(target_dir, filename, camera_name) File "/usr/lib/python2.7/site-packages/motioneye/uploadservices.py", line 103, in upload_file self.upload_data(rel_filename, mime_type, data, ctime, camera_name) File "/usr/lib/python2.7/site-packages/motioneye/uploadservices.py", line 357, in upload_data 'parents': [{'id': self._get_folder_id(path)}] File "/usr/lib/python2.7/site-packages/motioneye/uploadservices.py", line 401, in _get_folder_id folder_id = self._get_folder_id_by_path(location) File "/usr/lib/python2.7/site-packages/motioneye/uploadservices.py", line 413, in _get_folder_id_by_path parent_id = self._get_folder_id_by_name(parent_id, name) File "/usr/lib/python2.7/site-packages/motioneye/uploadservices.py", line 433, in _get_folder_id_by_name response = self._request(url) File "/usr/lib/python2.7/site-packages/motioneye/uploadservices.py", line 207, in _request self._credentials = self._request_credentials(self._authorization_key) File "/usr/lib/python2.7/site-packages/motioneye/uploadservices.py", line 287, in _request_credentials raise Exception(error.get('error_description') or error.get('error') or str(e)) Exception: Bad Request

One thing which may be irrelevant, but I'll say it anyway... I've noticed my Google Drive access key that got generated and entered in the motioneye authorization key did contain at least one backslash which I thought might be an 'unsafe' character unless fully escaped in code/transmission?

[BettySwallocks]

My uploadservices.json looks like this:

{ "1": { "gdrive": { "authorization_key": "4/1AX4XfWg2h1sYIqWbjvAQKoQUGT1Uy9unTMEsw4rktEU8GdTdxxHZVAkmeAg", "credentials": null, "location": "/loungeipcam/" } } }

The key seems to match what I got from Google security and entered on motioneye.

[BettySwallocks]

Can confirm after further testing this afternoon, such as resets of motioneye, rentry of camera and authorization details, plus clearing out Google Drive folders, I have compared to the OP text notes on Github and have exactly the same behaviours.

With motion detection ON - I get no images/videos uploaded to Google Drive, but it is able to create a folder under My Drive with the camera name (so points to not being a baseline authorization key issue).

Turning off motion detection (no other changes) allows manual snapshots to get uploaded to Google Drive without issue.

Turn on motion detection again (no other changes) and no further manual images or motion driven images/videos get uploaded to Google Drive.

[BettySwallocks]

Today I have seen images and video appear on my Google Drive with no changes made at my MotionEye or Home Assistant end. Perhaps an intermittent issue with Google Drive auth at their end?

My only clue is in my MotionEye log where I had been getting entries like: ERROR: gdrive: failed to obtain credentials: Bad Request

The errors stopped appearing after one last log message line that read: WARNING: accessing gdrive failed: Bad Request

I don't have times in my MotionEye log that's visible via my HA console, but it seems to tie in with when the Google Drive objects started to appear.

Hope the OP also sees some improvements and can confirm.

[jetmonk]

Hope the OP also sees some improvements and can confirm.

Unfortunately, I am away from my camera for an extended period.

[va1entin]

It seems like we'd need to implement a new OAuth flow as documented here. Google has a nice example with Flask that could be adapted for motioneye/tornado but there might be some pitfalls with the way uploadservices.py is implemented because the Google libs expect a certain flow and rely on some state for that.

However, I'd like to make a bigger proposal for motioneye's strategy: Remove the upload feature entirely; let dedicated tools do the job and provide guidance where it's needed.

Don't get me wrong: I've been using motioneye and the Google Drive upload specifically for many years and greatly missed it when it stopped working. I think uploading media files is an essential feature - I just don't think motioneye should do this itself.

There are widely used and stable tools like rclone that focus entirely on uploading, syncing, managing remote files. Using the "run a command" feature, one can easily let rclone run every time a media file is created and upload it to the remote destination that one desires.

To upload the current file without any folder structure one could use this command: rclone copy --verbose %f <rclone remote>:/<remote folder (optional)> --verbose is optional but prints some potentially interesting metrics to motion.log. The rclone remote must be set up before by using rclone config.

Imho the motioneye team/community should focus on providing how-to guides in the wiki or repo that explain how to set rclone up with some popular remotes (Google Drive, Dropbox, S3, ...). If something changes in a remote API, rclone will handle it and we "just" need to adapt the guide. This is still a task but should be much simpler than fixing custom upload logic.

By narrowing the scope of motioneye on its strengths, devs can focus on actually unique functionality instead of investing precious time into the moving target of remote storage APIs that tools like rclone already figured out for us. For me personally uploading files to remote storages is not a core feature or particular strength of motioneye and given the limited dev resources and availability of alternative tools a modular approach as outlined seems like the better way forward.

In case this feels like a loss of functionality, motioneye users could actually gain more functionality from this because rclone supports many more remotes than motioneye ever did. By providing guides that work across most or all of these, motioneye could suddenly support a plethora of remotes where there was a struggle to support even a very limited subset before.

@MichaIng I hope you don't mind me tagging you but I'd like to hear your view on this matter.

Having used rclone for many years, I'd happily contribute to the guides.