search

immense / Remotely

A remote control and remote scripting solution, built with .NET 8, Blazor, and SignalR.
GNU General Public License v3.0
4.28k stars 1.6k forks source link

Smooth upgrade and avoid breaking change. #865

Open Anduin2017 opened 3 months ago

Anduin2017 commented 3 months ago

I am a member of the data center operations team. Our data center heavily relies on Remotely for server management. We utilize a Docker Swarm cluster and have established a path from Ceph to the /remotely-data directory.

Recently, we received some security advisories indicating that we should always keep up with the latest image. Consequently, we configured automatic updates yesterday. However, just a few hours later, we started receiving numerous reports indicating that Remotely was no longer functioning. This led us to spend several hours investigating. On the release page, I noticed a "Breaking change" notification and discovered that the new version of Remotely no longer stores the database and configuration files under /remotely-data, but rather under /app/Remotely.db and /app/Data.

Due to these breaking changes, we attempted to copy the existing files to the new paths and then restart Remotely, but soon realized that the new version had different expectations regarding the database.


 Entered main script.
 Starting Remotely server.
 Unhandled exception. Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 1: 'no such table: DeviceScriptRun1'.
    at Microsoft.Data.Sqlite.SqliteException.ThrowExceptionForRC(Int32 rc, sqlite3 db)
    at Microsoft.Data.Sqlite.SqliteCommand.PrepareAndEnumerateStatements(Stopwatch timer)+MoveNext()
    at Microsoft.Data.Sqlite.SqliteCommand.GetStatements(Stopwatch timer)+MoveNext()
    at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
    at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
    at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader()
    at Microsoft.Data.Sqlite.SqliteCommand.ExecuteNonQuery()
    at System.Data.Common.DbCommand.ExecuteNonQueryAsync(CancellationToken cancellationToken)
 --- End of stack trace from previous location ---
    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteNonQueryAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteNonQueryAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteNonQueryAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
    at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationCommandExecutor.ExecuteNonQueryAsync(IEnumerable`1 migrationCommands, IRelationalConnection connection, CancellationToken cancellationToken)
    at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationCommandExecutor.ExecuteNonQueryAsync(IEnumerable`1 migrationCommands, IRelationalConnection connection, CancellationToken cancellationToken)
    at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationCommandExecutor.ExecuteNonQueryAsync(IEnumerable`1 migrationCommands, IRelationalConnection connection, CancellationToken cancellationToken)
    at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationCommandExecutor.ExecuteNonQueryAsync(IEnumerable`1 migrationCommands, IRelationalConnection connection, CancellationToken cancellationToken)
    at Microsoft.EntityFrameworkCore.Migrations.Internal.Migrator.MigrateAsync(String targetMigration, CancellationToken cancellationToken)
    at Program.<Main>$(String[] args) in D:\a\1\s\Server\Program.cs:line 291
    at Program.<Main>(String[] args)

Here, we wasted several hours and ultimately failed to migrate existing users to the new version of Remotely. We request that Remotely support a stable, smooth upgrade mode or an LTS channel, allowing deployers to enjoy the latest fixes and security updates while benefiting from the latest features.

As far as I know, Remotely uses Entity Framework, which inherently supports generating and managing migrations in situations involving multiple database sources and applying these migrations at application startup. This could address the issue of database breaking changes during upgrades. In fact, other applications we deploy, such as GitLab and Baget, support these features.

Frequent breaking changes are almost intolerable in large, fully automated data centers. Operators must frequently resort to manual interventions for a specific business.

Additionally, if smooth updates cannot be achieved, I still suggest providing clear export and import functionalities on the page. This way, before upgrading, we can export existing data, configurations, users, and servers into a format independent of the database and hardly subject to change, such as JSON or XML. After the upgrade is complete, we can use the import feature to migrate the core entities without losing data. This would be immensely beneficial for maintenance personnel.

At this moment, we are still struggling with upgrading from Docker Image 69 to Docker Image 88. It seems that abandoning existing data and reinstalling the Resilience Agent on each server is our only option.

Nonetheless, we appreciate your team. The functionality of Remotely itself greatly addresses our pain points.

bitbound commented 3 months ago

Hello and well met!

I can empathize with your situation, and I apologize for the frustrations. I think there are a few things I should explain about Remotely that users should know before relying on it in business environments.

There is no team. As you can see from the contributions graph, there is only me, the original author. I built Remotely as a hobby project while also working full time and raising a family. The original goal of Remotely was simply to manage a handful of computers for family and friends. Business scenarios were not part of my original design considerations.

The project ended up taking off in popularity. I tried to keep up with requests and to meet the growing list of user requirements, but I simply didn't have time or energy when work and family responsibilities were taking priority. I burnt out, multiple times.

Yet I came back to it a few times, hoping it might become a real revenue stream that could replace my full-time job. But in the process, I found myself neglecting my family, toiling away my nights and weekends that I should have been spending with my wife and kids. Eventually, though, I was able to drop it for good, which is when Darren from Immense/ImmyBot offered to buy it from me.

Most of the features in Remotely were done in haste, and usually while I was running on little sleep. I tried to build things with care and resilience in mind, but there's only so much a person can do in their spare time. There are only so many scenarios I can test in my home using my family's computers.

Had I somehow secured the funds to work on Remotely full-time without needing a second job, I'm confident I could have built a pretty decent RMM. I basically already have with my combined past work experience. 😆

the new version of Remotely no longer stores the database and configuration files under /remotely-data, but rather under /app/Remotely.db and /app/Data

The database is still stored wherever you want. The main change with the database is that the configuration is provided through environment variables instead of an appsettings.json file that needs to be mounted on the host. This was commonly requested and how it should have been done from the beginning. However, Remotely was originally only designed to be hosted on bare metal with dotnet and systemd, so I didn't consider these things in the beginning.

Breaking changes this severe are indeed undesirable. But I think in this case, it will be beneficial in the long run. And I'm hoping this is the last.

As far as I know, Remotely uses Entity Framework, which inherently supports generating and managing migrations in situations involving multiple database sources and applying these migrations at application startup.

You can see in that I'm already doing this, and I've been maintaining migrations for the three database providers. The migrations are applied automatically, and I've done my best to make them stable and painless. But again, there's only so much quality assurance I can do in my off hours. And there were a few times where I had to try to correct a mistake or poor design decision without knowing what kinds of weird states everyone's DB could be in as a result of those mistakes.

The table in the above error was dropped in a migration in August 2023. I could only guess at what's causing the error without being able to delve the data and environment.

I hope none of this comes across as my making excuses. Sometimes I make mistakes. Having limited time and resources to work on the project made it harder to catch them. I did the best I could.

I still suggest providing clear export and import functionalities on the page.

I agree that this would be a very helpful feature, especially for situations like this. However, I am no longer working on Remotely in my spare time, and most of my work time at ImmyBot is spent on other demands.

I can't speak for the company, but I'm guessing that any of my dev time that is allocated toward the project will be focused on remote control. Unless the community takes over development, the project is likely to remain pretty stable.

Sorry for the long rant. Hopefully it helps to explain why the project falls short of being a truly production-ready solution. Had I the time and resources, things would be different. 🙂

Cheers!

- Jared

Anduin2017 commented 2 months ago

Jared,

Thank you very much for your detailed response. I completely understand your situation. As a maintainer of an open-source project myself, I am all too familiar with the challenges of balancing regular work with project maintenance. Maintaining a project can be incredibly demanding and often lacks tangible rewards. That said, I must acknowledge that Remotely is indeed an exceptional solution for addressing various issues.

I have also opted to abandon all existing data and upgrade to version 88, reinstalling the agent on each of our client machines. While this process has been arduous, we have made the decision to stick with version 88, considering its stability and performance.

Wishing you every success in your entrepreneurial endeavors!