search

microsoft / Windows-Dev-Performance

A repo for developers on Windows to file issues that impede their productivity, efficiency, and efficacy
MIT License
434 stars 20 forks source link

OneDrive and Version Controlled and/or Source Code Directories #75

Open EricPell opened 3 years ago

EricPell commented 3 years ago

Environment

Item Value
OS, Version / Build All Windows 10 Builds
Processor Architecture AMD64/x86/ARM
Processor Type & Model AMD Ryzen 9 3950X/Irrelevant
Memory 64GB/Irrelevant
Storage Type, free / capacity (e.g. C: SSD 128GB / 512GB) C: NVMe 400GB/1TB, E: NVMe 600GB/1TB, D: SSD 800GB/ 1TB, Ext4 WSL: NVMe 900GB/ 1TB
Relevant apps installed OneDrive (Installed on OneDrive dedicated NVMe E:), VS Code, WSL, Git

Description

EDIT: No one is suggesting OneDrive should be used for source control. Read the post.

Among many other uses, OneDrive offers a chance to provide a synchronized developer environment on Windows (including subsystems like WSL), with a massive advantage that directories and projects can be synced without having to make (or undo) commits. This allows project development to happen on a multitude of devices, developers to "pick-up" their work without having to make temporary or incomplete commits. This makes cross-device development cleaner, more efficient and with less cluttered version control histories.

However, OneDrive does not function correctly with version controlled directories, like those with Git produce significantly reduced performance where directories contain many small files that can change rapidly. This also includes build directories for VS Projects was well as Linux compiled programs with WSL which exist within OneDrive. This can lead to constant syncing, and high resource usage by OneDrive.

These issues and possible solutions have been identified previously by users

  1. https://onedrive.uservoice.com/forums/913522-onedrive-on-windows/suggestions/18449926-reassess-the-possibility-to-ignore-or-exclude-sele
  2. https://onedrive.uservoice.com/forums/262982-onedrive-archive/suggestions/10932591-exclude-folders-inside-onedrive-folder-from-synchi

This issue is being raised here to provide a mechanism to allow MS project managers involved with Developer issues to raise the issue with the OneDrive team.

Steps to reproduce

  1. Use OneDrive on a large project with version control.
  2. Generate many small files, e.g. version controlled directories
  3. Try to sync an existing directory with many small files, e.g. version controlled directories

Expected behavior

One or all of the following behaviors would be expected:

First: It would be expected (hopefully) that OneDrive would sync these files efficiently. However, due to structural reasons this does not happen. So the expected mitigations would allow for using OneDrive in these directories with caveats.

  1. OneDrive will automatically identify such directories, if not all at least those controlled by popular version control software, like git, by using ".git" directories to flag them. These would provide the user a message NOT to place these in a OneDrive controlled folder.
  2. OneDrive will provide a means to disable syncing of said folders, for example with the presence of a .onedriveignore file in the root directory.
  3. Optimal solution would allow a convenient way to "archive" a build directory if needed, with paused syncing while building and cleaning. For example a command line api to OneDrive to pause and resume syncing.

Actual behavior

OneDrive syncs problematic folders without any accounting for the contents, which in the case of developer workloads can be problematic.

sylveon commented 3 years ago

Why not just put these kind of things outside OneDrive? Seems to me like the OneDrive app doing its job and syncing files as they come.

EricPell commented 3 years ago

Why not just put these kind of things outside OneDrive? Seems to me like the OneDrive app doing its job and syncing files as they come.

I have edited the text of the issue to make it more clear that the problem is syncing many small files, which is a know problem with OneDrive. I have also included a clear, but not exclusive statement about why this is advantageous to do in OneDrive. Final note, the posting of this issue was requested in order to allow it to be brought "up the chain" to the OneDrive team.

orcmid commented 3 years ago

Why not just put these kind of things outside OneDrive? Seems to me like the OneDrive app doing its job and syncing files as they come.

I have edited the text of the issue to make it more clear that the problem is syncing many small files, which is a know problem with OneDrive.

I use OneDrive but usually not for Code. The files of a coding class maybe, but not a code project. GitHub is far more useful for code efforts and no more expensive :).

One problem I have with OneDrive is that if I sync Music there, any playing of a song causes download of everything. GitHub has separate actions on small text files, and the related versioning, just right, the same as VSS in its day. The change to how syncing is handled seems to be an anti-pattern sometimes.

An effort to have OneDrive be one size that fits all strikes me as a bad idea. If I was gathering .git archives, I would put them on OneDrive as my backup of choice. But not the source control.

EricPell commented 3 years ago

An effort to have OneDrive be one size that fits all strikes me as a bad idea. If I was gathering .git archives, I would put them on OneDrive as my backup of choice. But not the source control.

No one is suggesting using OneDrive as source control.

I like using OneDrive to host git repositories so I can switch between my laptop and desktop, or work and home without have to make a temporary and messy commit. This was explained in the text of the post.

The issue is, generically, OneDrive has serious performance issues with any collection of small files.

zakius commented 2 years ago

serious performance issues with any collection of small files

seem to be a recurring topic for multiple tools

mikedon1 commented 1 year ago

My government organization doesn't allow us to use GitHub, but does have OneDrive. It would be very useful if we could use it with code version control.

trparky commented 1 year ago

I have found that executing GIT GC will clean up and compact all the various GIT versioning files that will help make syncing data via OneDrive much faster.

otri commented 1 year ago

OneDrive defaults to ON for a User's home environment; Documents, Downloads, etc.. Many users don't even realize this is the default in Windows 11.

And this sync behavior is especially problematic in an enterprise setting where OneDrive is the de-facto for a virtual desktop. If OneDrive can't handle git version control workloads, then it utterly craps on your only means of loading a virtual desktop environment.

OneDrive really needs to cooperate better with version control in a very deterministic way. For a known version controlled repo, it's probably safest to give the user a very clear option; for example, when trying to open a folder with .git hidden folder that's in conflict offer a resolution option.

GIT folder is in conflict with OneDrive remote, which would you like to keep?
  "Keep Local" or "Keep Remote"

This makes the option user driven, and make it easy to resolve the conflict only when the user needs to consider accessing the version controlled folder hierarchy. Cognitive load remains low, and is functionally compatible with how version control already works.