search

mutagen-io / mutagen

Fast file synchronization and network forwarding for remote development
https://mutagen.io
Other
3.44k stars 152 forks source link

Zero-length files with 0.12.0-beta2 in docker-compose #232

Closed f1sherman closed 2 years ago

f1sherman commented 4 years ago

Which version of Mutagen are you using (mutagen version)?

0.12.0-beta2

Which operating system (platform/version/architecture) are you using?

MacOS 10.15.6

What is the issue that you're experiencing?

When there is high I/O activity, some files have their contents zero'd out. It's sporadic/inconsistent, but happens often enough that it's noticeable.

What are the steps to reproduce the issue?

Any activity that creates a lot of I/O, autocorrecting linters in my case

What is the expected result of these steps in the absence of the issue?

The files do not have their contents zero'd out

What is the actual result?

The files have their contents zero'd out

Is there any other information that might be helpful?

I'm using the docker-compose integration. This did not happen on 0.12.0-beta1. Is it possible there were changes between the beta releases that could have caused this?

Thank you and please let me know what information I can provide that will help track this down!

xenoscopic commented 4 years ago

Hey @f1sherman, I'm sorry to hear about this, thanks for bringing it to my attention. Are these files having their contents zero'd out inside the container or on the local filesystem? Am I correct in assuming that the linters are running on the local system?

There weren't any major code changes in Mutagen's code between beta1 and beta2, but I did bump the Go compiler/runtime/standard library from 1.14.x to 1.15.x, which could definitely have some unintended consequences. I usually don't do major Go version upgrades during a release series, but I had assumed it would be okay in this case because of the relative maturity/stability of Go's toolchain/runtime/standard library and the fact that this was still in beta. Sorry :-(

Just a few more questions to help me reproduce/debug:

  1. Does this depend in any noticeable way on the size of the file being linted? (It shouldn't matter, the observable behavior should always be correct, but it might be easier to reproduce if I can find a file size that'll trigger things easier).
  2. Can you give me an example of the linter you're using? (It would help to understand the I/O strategy of the linker, how it's updating files, etc.)

I suspect this will require a deep-dive into the diff between Go 1.14 and 1.15 and that it'll be something really subtle. In the mean time, I can revert to Go 1.14 for the next beta release until this is sorted out. If you wouldn't mind testing that and letting me know if the issue continues, it would be a big help.

f1sherman commented 4 years ago

Thank you for the detailed response! One thing I neglected to mention is that the mutagen update happened as part of a brew upgrade so it's certainly possible that something other than mutagen is causing the issue.

Are these files having their contents zero'd out inside the container or on the local filesystem?

They end up zero'd out on both, I'm not sure which end initiates it though. Is there a log somewhere or debugging that I can turn on to help troubleshoot?

Does this depend in any noticeable way on the size of the file being linted?

I haven't noticed that. It has happened for relatively large yaml files during linting, I've also seen it happen with relatively small source code files when updating a code branch (updating the branch kicks off the linters, I haven't identified whether the zero-ing is caused by the git update or the linters, but I can turn them off and see if it still happens).

Can you give me an example of the linter you're using?

Rubocop is one of them. We also use some proprietary linters that I can't share for e.g. formatting yaml files.

Thanks again and please let me know what I can do to help troubleshoot the problem! Maybe I should try downgrading to beta1 just to confirm that the issue is in fact caused by the beta2 upgrade?

f1sherman commented 4 years ago

Hi there! I noticed today that I actually had what looked like 7 duplicate sync sessions. 6 of them had status "Halted due to one-sided root emptying" and the other had status "Watching for changes". I ended up doing a mutagen sync terminate --all and restarting my containers.

So far I haven't been able to reproduce the duplicate sync sessions. I'll keep an eye out and see if this issue comes back. Any chance the zero-length files issue I reported here might have been related? Happy to file a separate issue if that makes sense.

f1sherman commented 4 years ago

Knock on wood, the file truncation hasn't happened again since I terminated the duplicate sync sessions. So far the duplicate sessions haven't come back either. I'd be comfortable closing this and following up with y'all if I see it again, does that sound reasonable?

f1sherman commented 4 years ago

I've now seen the "duplicate sync sessions" issue crop up a few times. I believe it happens when docker-compose doesn't shut down gracefully. I've been able to reproduce it a couple of different ways:

  1. Restarting my host machine and then doing a docker-compose up
  2. Restarting Docker via the menu bar --> Restart and doing a docker-compose up

Any thoughts on how to mitigate this? Hopefully this is helpful!

xenoscopic commented 2 years ago

Circling back to this issue... (just a little late... sorry)...

I think you hit the nail on the head with the issue being duplicate sessions. This was a known issue with the way Mutagen tracked Compose-related sessions, which were (rather unfortunately) tied to the ephemeral Docker Desktop daemon ID, which could change on restarts, updates, and other events.

Mutagen's support for Compose V1 is now discontinued, so I'm going to close the issue. The new Compose-V2-based Mutagen Compose no longer uses this daemon identification mechanism, and thus shouldn't be susceptible to the same issue.