Local files not syncing in Jupyter Lab

[JesseRuttTX]

Hi, I am new to coiled and I am trying to run a basic cloud server from my mac just to test the sync feature. I have linked my GCP account and credentials to my coiled account and did all the necessary set up in my terminal and on GCP. I then run this command (coiled notebook start --sync) in my terminal and the server starts up flawlessly (see first screenshot). When the jupyter lab opens up though it is in a directory called "synced" but non of my folders and files are there (see second screenshot) and if I create folders or files they are not in my local directory after I shutdown the server. Is there something simple I am missing?

[ntabris]

Hi, @JesseRuttTX. Thanks for letting us know you're having problems with notebook sync. This doesn't appear to be any of the usual problems I've seen, so if you don't mind checking a couple things for me, that would be helpful for troubleshooting this.

In you notebook, you could try running !ls /scratch/synced and see if it lists any of your files. That's where the files actually sync to on the notebook server, and then it's symlinked to /tmp/synced (which is where jupyter is set to use as the working directory).

You might also check !ls -l /tmp to make sure the symlink is set as it should be:

Finally, in a local terminal (i.e., on the same computer where you ran coiled notebook start) you could run mutagen sync list and let me know what that shows. I'd expect to see something like this:

[JesseRuttTX]

Here is the output of running the "!ls /scratch/synced" and "!ls -l /tmp" command in an untitled notebook I created.

Here is the output of running the "mutagen sync list" command in a new local terminal window.

Here is what my directory looks like that I am trying to sync. It has like a few Gigabytes worth of data already in it. Is the default server able to handle syncing that much?

[JesseRuttTX]

Does this mean it is just taking a long time to sync?

[phofl]

It should be able to sync that much, but it can take quite some time based on your network connection. How fast is your upload speed?

I would probably recommend to load those files to remote storage and then pull them from there. That would ensure that you have to upload the data only once.

[ntabris]

@JesseRuttTX thanks for that info!

What's your working directory when you run coiled notebook start --sync? That's the directory that it's trying to sync, and from the screenshot of mutagen sync list it says the alpha (local) directory is all of your home directory (/Users/jessemac). If you haven't already tried, I'd try this after you cd in to that data directory.

That said, it's possible that even trying to sync a few GB won't be a great experience, mostly depending on the upload throughput you get from your local machine. You can run coiled notebook monitor <notebook name> to check the status of syncing. (The notebook name is shown at the top of the widget, e.g., notebook-0385e2d in your initial screenshot.)

For large data files, I'd recommend putting them in cloud storage and using them from there, rather than relying on the sync feature. In your case, that would probably mean using Google Cloud Storage. If you're not familiar with that and could use some guidance, let me know.

[JesseRuttTX]

it is still showing no sync files in my jupyter lab directory. Do they only show up there when they are all finished syncing?

[JesseRuttTX]

@JesseRuttTX thanks for that info!

What's your working directory when you run coiled notebook start --sync? That's the directory that it's trying to sync, and from the screenshot of mutagen sync list it says the alpha (local) directory is all of your home directory (/Users/jessemac). If you haven't already tried, I'd try this after you cd in to that data directory.

That said, it's possible that even trying to sync a few GB won't be a great experience, mostly depending on the upload throughput you get from your local machine. You can run coiled notebook monitor <notebook name> to check the status of syncing. (The notebook name is shown at the top of the widget, e.g., notebook-0385e2d in your initial screenshot.)

For large data files, I'd recommend putting them in cloud storage and using them from there, rather than relying on the sync feature. In your case, that would probably mean using Google Cloud Storage. If you're not familiar with that and could use some guidance, let me know.

Sorry i originally posted the wrong screenshot, i have edited the original comment with the correct screenshot. For now I will create a new directory to sync and pull my data from the cloud. That should solve my issue. Thank you for your help!