search

chervonij / DFL-Colab

DeepFaceLab fork which provides IPython Notebook to use DFL with Google Colab
1.05k stars 265 forks source link

Hourly Backups & Exports to Google Drive seem to fail #36

Closed nucleare closed 3 years ago

nucleare commented 3 years ago

Even though I've successfully connected my Google Drive and successfully uploaded my workspace.zip file

Mounted at /content/drive
/content
Done!

During training, the automatic backup seems to fail.

[04:14:41][#000785][9566ms][1.6624][1.9476]
[04:29:50][#000861][9324ms][1.6438][1.9358]
cp: failed to access '/content/drive/My Drive/': Transport endpoint is not connected
Backed up!
[04:46:43][#000934][3614ms][1.6339][1.9001]

I have tried modifying the code for training and added the force_remount tag to the training code but it still produces the same error during automatic hourly backup.

If I stop training and manually try to export my model folders and in the input field tried to rename the entry to "/workspacemanualback" I receive this error:

Mounted at /content/drive
/content/workspace
cp: cannot stat '/workspacemanualback': No such file or directory
rm: cannot remove '/workspacemanualback': No such file or directory
Done!

To see if renaming it caused an issue, i changed it back to workspace by default and the error is still receive the same error:

Mounted at /content/drive
/content/workspace
cp: cannot stat 'workspace': No such file or directory
rm: cannot remove 'workspace': No such file or directory
Done!

Although I have already cleared it from a previous session that failed and cant copy all the other errors in all it's other attempts to save or backup. this is what was shown for Get Result video

python3: can't open file 'DeepFaceLab/main.py': [Errno 2] No such file or directory
cp: cannot stat '/content/workspace/result.mp4': No such file or directory

Basically, I have yet to be able to use any session of Google Colab to help me with training but it's worth noting that I am only uploading my workspace AFTER i've done the extraction and some masking for Xseg in advance, which means my workspace.zip is 10gb+ while the src and dst video files are 2gb and 6gb respectively

I have updated to Colab Pro and still cant seem to export any work that is done on Colab, any ideas? Some notes that may or may not help:

Is there any other ways to pull/download the data off the notebook aside from exporting to Google Drive that you know if?

nucleare commented 3 years ago

The issue was discovered and was particular to the environment, the code from this repository has been checked and is unaffected and therefore does not represent a universal bug. Somehow the coding for the mount director in Google drive was changed in the Training code side instructions.