cguZZman / plugin.googledrive

The Google Drive addon for Kodi
GNU General Public License v3.0
213 stars 72 forks source link

Faster strm exports #320

Open 0o120 opened 1 year ago

0o120 commented 1 year ago

Google Drive Test Directory:

Export Directory:
    /Movies/

Contents:
    /Movies/Movie1/Movie1.mp4
    /Movies/Movie2/Movie2.mp4
    /Movies/Movie3/Movie3.mp4
    ... 2,292 total movies

Before Changes Google Drive API Requests: 2317 Export Duration: 15 minutes and 40 seconds (940 seconds)

After Changes Google Drive API Requests: 28 Export Duration: 4 minutes and 26 seconds (266 seconds)

Could be faster, but I think it's slowed down with the way clouddrive is handling each google drive item. This is what I could do with minimal changes.

It fetches all the files\folders in the directory with minimal api requests and then just feeds it back to clouddrive with the providers instance cache.

0o120 commented 1 year ago

Was able to get that same 266 second export down to 47 seconds by disabling the export_manager database access inside the while loop located inside clouddrive.common.service.export.ExportService.process_pending_changes and saving the items after the loop.

You can see in these changes: https://github.com/0o120/script.module.clouddrive.common-1/commit/c5a5b7b7c10dff96992cc559fff06718ac62d8c2

I'm assuming export_manager.save_pending_changes() is used to save progress in the event of an interruption and not exactly necessary? Maybe adding a config option to disable save_pending_changes inside that loop in exchange for performance gain?

I'm up for adding a config option. Some thoughts on this would be great :)

cguZZman commented 7 months ago

Thanks for the improvement. It's been a long time and this project is been in KTLO mode for me. But I will merge your change if you confirm it's been working fine for you. Also, saving the progress is necessary for large catalogs. If you submit your other change with a config option, I could merge it too.