GoPro Upload Issue: Excessive Requests, Directory Restructuring, and Non-existent File Errors

[NebulusIO]

Description:

I have been using an application to upload a large number of files from various sources to the GoPro cloud. This application has been a godsend, helping me consolidate and upload files that were scattered across multiple clouds and drives, some of which I was unsure if I had already uploaded to the GoPro cloud. However, I have encountered several issues and have had to create scripts in Haskell to support my use case and extend the functionality of the application.

Background:

I have been trying to rectify a lack of system organization dating back to 2017. I have a large number of files and exports scattered across different clouds and drives, and I am trying to ensure that all these files are backed up to the GoPro cloud for peace of mind in case of HDD failure.

Enhancements Made:

1. Developed a Haskell script to automate the GoPro upload process from a shell script on Mac. The script notifies the user upon completion and triggers an alert that requires user interaction to finish, ensuring a smooth workflow.
2. Created a validation script to ensure upload paths do not contain strings, which can be problematic when sourcing from Google Drive due to the unremovable space in "My Drive".
3. Wrote a script to flatten directory's nested folders, addressing the issue of previously unexplainable organization methods.
4. Developed a script to partition a directory into subdirectories of 30 files each, as GoPro seems to prefer this number and it matches the maximum number of notifications shown on the status upload page when a duplicate is encountered.
5. Extended GoPro upload functionality with a script that accepts nested directories and executes uploads on each directory sequentially.
6. Implemented a sync script to log out to a specified directory.

Current Problem:

After restructuring directories to flatten all files and then partition them back sequentially by filename into chunks of 30, I encountered an issue. The GoPro upload command is failing with an error stating that the file it's trying to finish does not exist. This is presumably because the file was moved or deleted during the partitioning process. This issue arose when I attempted to upload around 250 files out of 4TB of footage.

Potential Solution:

I am considering running the GoPro cleanup command to stop expecting uploads from the desktop. However, I am concerned that this might also stop and close files that have finished the uploading phase and are still in processing. A safer alternative might be to inform the server to stop waiting for an upload that is no longer coming, while ensuring that the server continues to process and finish any files that are currently in a processing state. I am looking for confirmation or a workaround for this issue.

[dustin]

Heh, the idea of a script in Haskell is humorous to me. We have some of those at work, though.

The cleanup command only cleans up things in "registered", "uploading", or "failure" state. It's a thing I use when I mess up an upload or something, but shouldn't affect anything that (probably) uploaded correctly.

i.e., I think cleanup does what you want done. It will not touch any transcoding media and will tell you what it cleaned up and what state that thing was in.

[NebulusIO]

Surprisingly simpler than I expected compared to strict bash. I wanted to try something different than python. That must be functional programming for you.

---

Appreciate the clarification, docs eluded to that but wasn't certain.

---

Another thing, is it normal to see the following log indefinitely?

gopro sync --verbose D: Reading auth token from DB I: 0 new items I: Fetching meta 0 D: Need meta: [] I: Need to fetch 10 moments

How do I find what moments it keeps trying to fetch?

[dustin]

Surprisingly simpler than I expected compared to strict bash. I wanted to try something different than python. That must be functional programming for you.

Yeah, I suppose I don't write many "scripts" these days in the classical sense of throwing things together in a scripting language.

Appreciate the clarification, docs eluded to that but wasn't certain.

Good point, I added a bit to this.

Another thing, is it normal to see the following log indefinitely?

gopro sync --verbose D: Reading auth token from DB I: 0 new items I: Fetching meta 0 D: Need meta: [] I: Need to fetch 10 moments

How do I find what moments it keeps trying to fetch?

Interesting, I've not seen that. The actual query it's running (assuming SQLite) is:

select m.media_id from media m left outer join
  (select media_id, count(*) as moco from moments group by media_id) as mo
  on (m. media_id = mo.media_id)
 where m.moments_count != ifnull(moco, 0) ;

That will give you the list of ones it thinks have moments that aren't recorded locally.

TBH, I don't use the moments feature all that much, but I do hit the button now and then. I've got ~185 in my DB.