files under $HOME were recursively deleted when monarch upgrade

[yagi2]

I did a monarch upgrade in shell to upgrade monarch.

In doing so, it seems to have failed to remove binary that already exsisted. Here is the log. ( some parts are masked with dummies.)

 !  ~/r/g/e/xxx  branch-name     monarch upgrade                                                                                                                                              2.6m  Tue Mar  8 07:44:31 2022

## Join our newsletter
Sign up to receive low frequency emails on the latest Monarch updates, 
features and news!

Enter email (optional): 

## Stay in touch
- GitHub: https://github.com/Dropsource/monarch
- Twitter: https://twitter.com/monarch_app
- YouTube: https://www.youtube.com/channel/UCuG4P4KTQ2HbikL57HIc1OQ
- Newsletter: https://monarchapp.io/docs/community

## Upgrading Monarch
New Monarch version will be installed in /Users/yagi2

Downloading Monarch installation bundle version 1.7.3

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2885k  100 2885k    0     0  1367k      0  0:00:02  0:00:02 --:--:-- 1379k

Extracting installation bundle...Done
Deleting pre-existing Monarch binaries...Error deleting pre-existing Monarch binaries. Monarch may need to be re-installed.
yagi2@hostname ~/r/g/e/xxx [124]>   

So far so good, but after this command completed, files under $HOME ( /Users/yagi2 ) were deleted recursively. You can see in the first and last line of the log above that the shell configuration has been deleted.

Is there an implemenation that deletes anything other than the main body of the monarch binary? ( inDeleting pre-existing Monarch binaries... phase) should check this implementation immediately. (I tried to find the implementation in this repository, but could not find where the monarch upgrade is implemented.)

[yagi2]

I installed monarch binary in /Users/yagi2/bin/monarch before enter upgrade command. Of course, monarch upgrade also removed the /Users/yagi2/bin directory, so the binaries are gone.

[fertrig]

The upgrade command is supposed to only delete the monarch directory. Can you see if you can get the log file for the upgrade command?

Open the file: /Users/yagi2/Library/Application Support/com.dropsource.monarch/logs/logs.info

You should be able to see a log entry with [`monarch upgrade`]. That log entry should have the location of the temp log file.

Could you please send me that log file? Post it here or email it to ftrigoso@dropsource.com

[yagi2]

in /Users/yagi2/Library/Application Support/com.dropsource.monarch/logs/logs.info

2022-03-08 07:44:34.694414 [`monarch upgrade`] [/var/folders/29/py4d162s55vfp1yrr0046dnm0000gn/T/monarch_cli_apNcbD/log_monarch_cli.log]

in /var/folders/29/py4d162s55vfp1yrr0046dnm0000gn/T/monarch_cli_apNcbD/log_monarch_cli.log

2022-03-08 07:44:34.689460 CONFIG [ContextInfo] ...
2022-03-08 07:44:34.694294 CONFIG [ContextInfo] User device id read successfully, user_device_id=426daaee-a401-478f-a5c2-02a34f13e03c
2022-03-08 07:44:34.733681 CONFIG [ContextInfo] Operating system information, name=macos version="12.2" build_version=21D49 
2022-03-08 07:44:34.947872 CONFIG [ContextInfo] xcodebuild info, xcode_version=13.2.1 xcode_build_version=13C100
2022-03-08 07:44:34.948260 CONFIG [SessionManager] session_id=3c30dcda-a0b4-4cad-96be-4c4e1d1e0eaa
2022-03-08 07:44:34.948526 INFO [StandardOutput] ## Join our newsletter
Sign up to receive low frequency emails on the latest Monarch updates, 
features and news!
2022-03-08 07:44:34.948552 INFO [StandardOutput] Enter email (optional):
2022-03-08 07:44:37.353793 INFO [StandardInput] 
2022-03-08 07:44:37.356650 INFO [StandardOutput] ## Stay in touch
- GitHub: https://github.com/Dropsource/monarch
- Twitter: https://twitter.com/monarch_app
- YouTube: https://www.youtube.com/channel/UCuG4P4KTQ2HbikL57HIc1OQ
- Newsletter: https://monarchapp.io/docs/community
2022-03-08 07:44:37.356878 INFO [StandardOutput] ## Upgrading Monarch
2022-03-08 07:44:37.356895 INFO [StandardOutput] New Monarch version will be installed in /Users/yagi2
2022-03-08 07:44:37.959329 CONFIG [ContextInfo] >>>
2022-03-08 07:44:38.182440 INFO [Upgrader] Using temp directory /var/folders/29/py4d162s55vfp1yrr0046dnm0000gn/T/monarch_upgrade_iE27Ww
2022-03-08 07:44:38.182442 INFO [StandardOutput] 
2022-03-08 07:44:38.182459 INFO [StandardOutput] Downloading Monarch installation bundle version 1.7.3
2022-03-08 07:44:38.182470 INFO [StandardOutput] 
2022-03-08 07:44:38.227551 FINE [Downloader]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2022-03-08 07:44:38.426821 FINE [Downloader] 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2022-03-08 07:44:39.532366 FINE [Downloader] 
  3 2885k    3  109k    0     0  84659      0  0:00:34  0:00:01  0:00:33 85868
2022-03-08 07:44:40.319602 FINE [Downloader] 
100 2885k  100 2885k    0     0  1367k      0  0:00:02  0:00:02 --:--:-- 1379k

2022-03-08 07:44:40.321950 FINE [Downloader] command="curl -O https://d2dpq905ksf9xw.cloudfront.net/macos/monarch_macos_1.7.3.zip" exit_code=0 is_success_code=true is_terminated=false
2022-03-08 07:44:40.321954 INFO [StandardOutput] 
2022-03-08 07:44:40.321972 INFO [StandardOutput] Extracting installation bundle...
2022-03-08 07:44:40.393832 FINE [Unzipper] command="unzip -q monarch_macos_1.7.3.zip" exit_code=0 is_success_code=true is_terminated=false
2022-03-08 07:44:40.393839 INFO [StandardOutput] Done
2022-03-08 07:44:40.393863 INFO [StandardOutput] Deleting pre-existing Monarch binaries...
2022-03-08 07:44:40.394833 FINE [Upgrader] Deleting /Users/yagi2/.config
2022-03-08 07:44:40.394834 FINE [Upgrader] Deleting /Users/yagi2/Music
2022-03-08 07:44:40.394835 FINE [Upgrader] Deleting /Users/yagi2/repos
2022-03-08 07:44:40.394835 FINE [Upgrader] Deleting /Users/yagi2/.docker
2022-03-08 07:44:40.394835 FINE [Upgrader] Deleting /Users/yagi2/.dart
2022-03-08 07:44:40.394836 FINE [Upgrader] Deleting /Users/yagi2/.DS_Store
2022-03-08 07:44:40.394836 FINE [Upgrader] Deleting /Users/yagi2/.CFUserTextEncoding
2022-03-08 07:44:40.394836 FINE [Upgrader] Deleting /Users/yagi2/.wget-hsts
2022-03-08 07:44:40.394836 FINE [Upgrader] Deleting /Users/yagi2/bin
2022-03-08 07:44:40.394837 FINE [Upgrader] Deleting /Users/yagi2/.rbenv
2022-03-08 07:44:40.394837 FINE [Upgrader] Deleting /Users/yagi2/.local
2022-03-08 07:44:40.394837 FINE [Upgrader] Deleting /Users/yagi2/Creative Cloud Files
2022-03-08 07:44:40.394838 FINE [Upgrader] Deleting /Users/yagi2/Pictures
2022-03-08 07:44:40.394838 FINE [Upgrader] Deleting /Users/yagi2/.dartServer
2022-03-08 07:44:40.394840 FINE [Upgrader] Deleting /Users/yagi2/.pub-cache
2022-03-08 07:44:40.394840 FINE [Upgrader] Deleting /Users/yagi2/.zsh_history
2022-03-08 07:44:40.394841 FINE [Upgrader] Deleting /Users/yagi2/Desktop
2022-03-08 07:44:40.394841 FINE [Upgrader] Deleting /Users/yagi2/Library
2022-03-08 07:44:40.394841 FINE [Upgrader] Deleting /Users/yagi2/.emulator_console_auth_token
2022-03-08 07:44:40.394841 FINE [Upgrader] Deleting /Users/yagi2/.android
2022-03-08 07:44:40.394842 FINE [Upgrader] Deleting /Users/yagi2/.cocoapods
2022-03-08 07:44:40.394842 FINE [Upgrader] Deleting /Users/yagi2/.cups
2022-03-08 07:44:40.394842 FINE [Upgrader] Deleting /Users/yagi2/Public
2022-03-08 07:44:40.394843 FINE [Upgrader] Deleting /Users/yagi2/.hyper_plugins
2022-03-08 07:44:40.394843 FINE [Upgrader] Deleting /Users/yagi2/.gitignore
2022-03-08 07:44:40.394843 FINE [Upgrader] Deleting /Users/yagi2/.flutter
2022-03-08 07:44:40.394843 FINE [Upgrader] Deleting /Users/yagi2/xxx
2022-03-08 07:44:40.394844 FINE [Upgrader] Deleting /Users/yagi2/.ssh
2022-03-08 07:44:40.394844 FINE [Upgrader] Deleting /Users/yagi2/StudioProjects
2022-03-08 07:44:40.394844 FINE [Upgrader] Deleting /Users/yagi2/Movies
2022-03-08 07:44:40.394846 FINE [Upgrader] Deleting /Users/yagi2/Applications
2022-03-08 07:44:40.394846 FINE [Upgrader] Deleting /Users/yagi2/.gradle
2022-03-08 07:44:40.394847 FINE [Upgrader] Deleting /Users/yagi2/.flutter-devtools
2022-03-08 07:44:40.394847 FINE [Upgrader] Deleting /Users/yagi2/.Trash
2022-03-08 07:44:40.394847 FINE [Upgrader] Deleting /Users/yagi2/.git_template
2022-03-08 07:44:40.394848 FINE [Upgrader] Deleting /Users/yagi2/github.com
2022-03-08 07:44:40.394848 FINE [Upgrader] Deleting /Users/yagi2/.npm
2022-03-08 07:44:40.394848 FINE [Upgrader] Deleting /Users/yagi2/Documents
2022-03-08 07:44:40.394848 FINE [Upgrader] Deleting /Users/yagi2/OneDrive - xxx
2022-03-08 07:44:40.394849 FINE [Upgrader] Deleting /Users/yagi2/.vscode
2022-03-08 07:44:40.394849 FINE [Upgrader] Deleting /Users/yagi2/.hyper.js
2022-03-08 07:44:40.394849 FINE [Upgrader] Deleting /Users/yagi2/.swiftpm
2022-03-08 07:44:40.394849 FINE [Upgrader] Deleting /Users/yagi2/.esets
2022-03-08 07:44:40.394850 FINE [Upgrader] Deleting /Users/yagi2/Downloads
2022-03-08 07:44:40.394850 FINE [Upgrader] Deleting /Users/yagi2/tmp
2022-03-08 07:44:40.394850 FINE [Upgrader] Deleting /Users/yagi2/.gitconfig
2022-03-08 07:44:40.394852 FINE [Upgrader] Deleting /Users/yagi2/.tig_history
2022-03-08 07:44:40.394852 FINE [Upgrader] Deleting /Users/yagi2/fvm
2022-03-08 07:44:40.394852 FINE [Upgrader] Deleting /Users/yagi2/.zsh_sessions
2022-03-08 07:46:05.495049 SEVERE [Upgrader] Error deleting pre-existing monarch files
Error or Exception details:
FileSystemException: Deletion failed, path = '/Users/yagi2/Public' (OS Error: Permission denied, errno = 13)
2022-03-08 07:46:05.495082 INFO [StandardOutput] Error deleting pre-existing Monarch binaries. Monarch may need to be re-installed.

The logs appear to show that everything has been deleted. All the projects I was working on and other things disappeared. ......

[yagi2]

environment values when I entered monarch upgrade .

$ monarch --version                                                                                                                                                                        Monarch binaries version: 1.6.0
Monarch CLI: v1.6.14
Monarch UI: v1.2.3
Operating system: macos

[fertrig]

We are so sorry this happened to you. We will find a way to compensate you.

The issue seems to only happen if the monarch\bin directory is moved out of the monarch directory. It is a bad bug. We will fix it asap.

[fertrig]

Fix released with Monarch 1.7.4.

[divan]

This just deleted my all HOME directory files too 😩

I can't even assess at the moment the amount of damage it created. All dot directories and files were wiped out too.

I don't remember anything like this has ever happened to me in 20+ years. :(

[Esc4iCEscEsc]

@fertrig I'm trying to see where the bug was initially and also how solving the issue was approached. Can you link me the correct commit where it was fixed, as I can't seem to find it?

The 1.7.4 release reads as follows currently:

The monarch upgrade command used to delete old monarch files. Under certain conditions, a bug manifested where the command could delete non-monarch files.

Do you think maybe this is under-selling the bug? There is no mention of peoples $HOME directory being wiped clean, nor am I seeing any big warnings from your side about deleting peoples files in any communication from either you nor @Dropsource.

[fertrig]

We are sorry this happened again.

The issue only happens if the monarch\bin directory is moved out of the monarch directory. It is not something most users would do.

The fix doesn't delete anything anymore so it is safe.

The fix is in a project we are working on open sourcing. Thus it is closed source right now.

I will talk to the rest of our team on Monday to discuss a community announcement.

Again, our apologies.

[Esc4iCEscEsc]

@fertrig even if monarch isn't 100% open source, could you maybe publish the git .patch for the fix immediately instead of waiting for it?

Understandable that the issue doesn't happen anymore in the newer versions, but that (as evident by the comment by @divan) doesn't mean it's been fixed in all versions.

The issue only happens if the monarch\bin directory is moved out of the monarch directory. It is not something most users would do.

Are you saying the issue happens anytime the monarch binary isn't inside of a directory called bin or what exactly is the scenarios you're talking about here? Understanding the risks would be much easier if you could just show the diff of the issue getting fixed, even if the entire source code behind the updates is not open source.

I always move binaries into my own directories (often ~/.local-bin which I have manually added to $PATH) but it's unclear if this is what you are referring to or not. I also am not alone in doing this with binaries (moving them, not specifically to a .local-bin directory).

I hope more people don't come across this issue until Monday, as you are willingly waiting to address this issue further.

[tyler-smith]

@fertrig

The fix doesn't delete anything anymore so it is safe.

This doesn't help users who have the broken version already. When they try to upgrade to the fixed version they'll wipe their HOME. You should actively communicate to your users to ensure they haven't moved the bin and how to upgrade safely to the patched version.

[divan]

I use HOME/bin for everything that is installed not via brew/app store.

Monarch is a really nice tool (I love it, really), but it's not something that's used daily. Many people upgrade existing tools not on a regular basis, but maybe once a month or even once a half a year. I can expect many users have now old version, and aren't aware that upgrade can ruin their dev workstation.

If there is any way to communicate that they should not run 'monarch upgrade' that can prevent damage.

[fertrig]

We may be able to reject any upgrade requests in our remote API which will prevent the upgrade process from running, which will prevent the bug from manifesting.

Working on this now.

[fertrig]

Change deployed. All upgrade requests will be rejected by the remote API, which stops the upgrade process from running locally.

Any users that run monarch upgrade will see "Monarch API returned non-successful status code" and the process will stop.

To get the new monarch version, you can do it manually as if it was your first time installing monarch: https://monarchapp.io/docs/install

We'll send out notifications to Monarch users early next week with more details.

[tyler-smith]

Thanks for the quick response today. Perhaps you can make the update API request send the current version (that is being upgraded from) so you can continue to provide the automated update as long as they're >= the fixed version.

[fertrig]

Yes, you are correct, that's what I have in mind as well.

[fertrig]

The new monarch release includes improvements to the upgrade process: https://monarchapp.io/blog/version-1.7.6

[Esc4iCEscEsc]

Last weekend, some users reported a severe bug in the upgrade process which was deleting local files. Within two hours of the report, we changed our remote API so upgrade requests would be rejected

@fertrig really? The history of this very GitHub issue is public and has timestamps. You received the first report of this happening 16 days ago (March 7th) and disabled it 12 days later (March 19th), not two hours after the first report. Why is it so hard for Dropsource/you to be honest about this issue? I realize it's embarrassing for you, but you're not giving out a lot of confidence by 1) not acknowledging the full scope of the issue and 2) not being honest about the response.

I frankly love the UX of Monarch, but how you all have dealt with this issue has removed any trust I had that you know what you're doing.

[fertrig]

You are right that it was reported days before but we did provide a fix back then. However, we also failed to realize the scope of the problem. Last weekend that was made clear and we fixed it two hours later.

I was out of town on a weekend trip when this happened and I stopped what I was doing to fix it. I have also been working non stop for the past three days on a good improvement and fix.

We take this product very seriously.

We didn't mean to hide or be dishonest about anything. I understand how you perceived it that way though. However, that was not my intention at all.

It is the opposite actually. I'm pushing to make all of Monarch open source. We want to be more open and transparent about what we do.

This is just the beginning for Monarch. We have many major improvements in the pipeline. We will learn from this mistake and improve our processes.

A lot goes on behind the scenes in Monarch to provide the experience you love.

We will continue to work hard to regain the trust users like you have lost.