Scripts to update config/overleaf.rc and config-seed rebrand

mserranom commented 7 months ago

Description

Updates variables config-seed/overleaf.rc with OVERLEAF_ prefix, and replaces occurrences of those variables through the repo.
Updates config-seed/variables.env, and updates the documentation accordingly.
Adds scripts to upgrade users' config/overleaf.rc by extending the scripts added in https://github.com/overleaf/toolkit/pull/214

This PR needs to be merged once CE/SP 5.0.0 is shipped, contains the changes to config-seed/version and CHANGELOG. The latter needs to be amended with the release date and release note links.

Related issues / Pull Requests

Contributor Agreement

[x] I confirm I have signed the Contributor License Agreement

mserranom commented 7 months ago

I pushed some changes to check config/overleaf.rc without checking config/version first and tested some scenarios

Run an existing 4.x instance in this branch

✗ bin/up -d
WARNING: some runtime variables defined in config/overleaf.rc
contain 'SHARELATEX_' in their name. The variables should be
renamed to 'OVERLEAF_'.
you can upgrade your config/overleaf.rc by running bin/rename-rc-vars
Do you want to continue? (y/n): n
Exiting.

✗ bin/rename-rc-vars
This script will update your config/overleaf.rc.
We recommend backing up your config with bin/backup-config.
Do you want to continue? (y/n): y
Found 5 lines with SHARELATEX_ in overleaf.rc
Creating backup file config/__old-overleaf.rc.2024.02.13-12.31.10
Replacing 'SHARELATEX_' with 'OVERLEAF_' in config/overleaf.rc
Updated 5 lines in /Users/m.serrano/workspace/toolkit/config/overleaf.rc
Done.

✗ bin/up -d

✗ curl http://localhost
Found. Redirecting to /login%

Tested:

Create edit, compile project
Navigate through history

Upgrade from 4.x to 5.0.0-RC

✗ cat config/version
4.2.0
✗ echo 5.0.0-RC > config/version
✗ cat config/version
5.0.0-RC

✗ bin/up -d
WARNING: some environment variables defined in config/variables.env
contain 'SHARELATEX_' in their name
Starting with Overleaf CE and Server Pro 5.0.0 the environment variables
use the prefix 'OVERLEAF_'  instead of 'SHARELATEX_'.
Please check your config/version, and config/variables.env files,
you can upgrade your config/variables.env by running bin/rename-env-vars-5-0
Do you want to continue? (y/n): n
Exiting.

✗ bin/rename-env-vars-5-0
This script will update your config/variables.env and config/overleaf.rc.
We recommend backing up your config with bin/backup-config.
Do you want to continue? (y/n): y
Found 60 lines with SHARELATEX_ in variables.env
Creating backup file config/__old-variables.env.2024.02.13-13.06.00
Replacing 'SHARELATEX_' with 'OVERLEAF_' in config/variables.env
Updated 60 lines in /Users/m.serrano/workspace/toolkit/config/variables.env
Done.

✗ bin/up -d

✗ curl http://localhost
Found. Redirecting to /login%

mserranom commented 7 months ago

I've added a couple of changes after a conversation with @das7pad, rebased and worked out a better commit breakdown to facilitate the code review (lost some attribute, apologies).

mserranom commented 7 months ago

For users who git pull before running bin/upgrade, it might be worth adding a check in bin/docker-compose to refuse to work when the rc file has not been migrated yet. Otherwise custom ..._DATA_PATH would not be picked up.

That's a good point. I think moving the checks from bin/up to bin/docker-compose will do the thing 👍🏻

mserranom commented 7 months ago

For users who git pull before running bin/upgrade, it might be worth adding a check in bin/docker-compose to refuse to work when the rc file has not been migrated yet. Otherwise custom ..._DATA_PATH would not be picked up.

That's a good point. I think moving the checks from bin/up to bin/docker-compose will do the thing 👍🏻

Done in https://github.com/overleaf/toolkit/pull/217/commits/eeb9ae3d827039f289b0fc361060b5c868166311

mserranom commented 7 months ago

It turns SHARELATEX_MONGO_URL and SHARELATEX_REDIS_HOST where only working in <v4 because I forgot to remove the original variables 🤦🏻 (https://github.com/overleaf/toolkit/pull/217/commits/f67fae9407fa31d1e020be6ee1244f9abedb3589)

I've ended up extracting the variables into override compose files in https://github.com/overleaf/toolkit/pull/217/commits/0dd9750a5870a825abc885329e7401e9602f2a65. Since we already have similar mechanisms in place I found it a better option than running substitutions in docker-compose.base.yml. It's tested in 4.2.3 and the nightly build tagged as 5.0.0-RC.

das7pad commented 7 months ago

WDYT about performing the RC migration before running handle_image_upgrade?

Done via be3b10a1821a1e3f88cb5038582d308dd46d5d20

I've also added a prompt before migrating the overleaf.rc: 7d2496e2eb9c4b5cdbf73cb2d1fae801c4448eb6

bin/docker-compose/ bin/up now refuses to start when the branding of variables does not match the version dcc19a781e5054b39dc3dd5c98775a2e478cdf12. I've checked for any OVERLEAF_ branded in the monorepo prior to our rebanding and did not find any items that could yield false-positives for customers. If someone really wants to override this, they can remove the check.

Then there are a few copy changes to make the output more readable, e.g.

# ShareLaTeX rc
toolkit$ bin/up
Initiating Mongo replica set...
Rebranding from ShareLaTeX to Overleaf
  The Toolkit has adopted to Overleaf brand for its variables.
  Your config/overleaf.rc still has 3 variables using the previous ShareLaTeX brand.
  Moving forward the 'SHARELATEX_' prefixed variables must be renamed to 'OVERLEAF_'.
  You can migrate your config/overleaf.rc to use the Overleaf brand by running:

    toolkit$ bin/rename-rc-vars

# ShareLaTeX variables and 5.x
toolkit$ bin/up
Initiating Mongo replica set...
Rebranding from ShareLaTeX to Overleaf
  Starting with Overleaf CE and Server Pro version 5.0.0 the environment variables will use the Overleaf brand.
  Previous versions used the ShareLaTeX brand for environment variables.
  Your config/variables.env has 45 entries matching 'SHARELATEX_', expected prefix 'OVERLEAF_'.
  Please align your config/version with the naming scheme of variables in config/variables.env.
  You can migrate your config/variables.env to use the Overleaf brand by running:

    toolkit$ bin/rename-env-vars-5-0

# Overleaf variables and 4.x
toolkit$ bin/up
Initiating Mongo replica set...
Rebranding from ShareLaTeX to Overleaf
  Starting with Overleaf CE and Server Pro version 5.0.0 the environment variables will use the Overleaf brand.
  Previous versions used the ShareLaTeX brand for environment variables.
  Your config/variables.env has 45 entries matching 'OVERLEAF_', expected prefix 'SHARELATEX_'.
  Please align your config/version with the naming scheme of variables in config/variables.env.

And announcing that the migration steps will take a backup, just before prompting the user to confirm the operation:

$ bin/rename-env-vars-5-0
This script will update your config/variables.env.
We recommend backing up your config with bin/backup-config.
Rebranding from ShareLaTeX to Overleaf
  Found 45 lines with SHARELATEX_ in config/variables.env
  Will create backup of config/variables.env at config/__old-variables.env.2024.02.26-16.49.32
  Proceed with the renaming in config/variables.env? (y/n): y
  Creating backup file config/__old-variables.env.2024.02.26-16.49.32
  Replacing 'SHARELATEX_' with 'OVERLEAF_' in config/variables.env
  Updated 45 lines in config/variables.env
Done.

(This will be printed by bin/upgrade as well)

bin/upgrade

``` # Setup: checkout jpa-msm-overleaf-rc-rebrand, then reset to e2cbe387e032dfc2757133c1f421b97e3f7e287e aka https://github.com/overleaf/toolkit/pull/225 to get the re-launching of bin/upgrade after making changes to it toolkit$ bin/upgrade Warning: current branch is not master, 'jpa-msm-overleaf-rc-rebrand' instead Checking for code update... No code update available for download Changelog: ---------- +## TODO +### Added +- Updated default [`version`](https://github.com/overleaf/toolkit/blob/master/lib/config-seed/version) to `5.0.0`. + + :warning: This is a major release. Please check the [release notes](TODO) for details. + +- Rebranded 'SHARELATEX_' variables to 'OVERLEAF_' + ---------- Perform code update? [y/n] y Current commit is e2cbe38 Updating code... From github.com:overleaf/toolkit * branch jpa-msm-overleaf-rc-rebrand -> FETCH_HEAD Updating e2cbe38..38c387d Fast-forward CHANGELOG.md | 8 ++++++++ bin/docker-compose | 33 +++++++++++++++++++++++---------- bin/doctor | 101 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------- bin/rename-env-vars-5-0 | 8 +++----- bin/rename-rc-vars | 42 ++++++++++++++++++++++++++++++++++++++++++ bin/up | 17 ----------------- bin/upgrade | 12 ++++++++---- doc/docker-compose.md | 2 +- doc/ldap.md | 21 +++++++++++---------- doc/overleaf-rc.md | 14 +++++++------- doc/persistent-data.md | 2 +- doc/saml.md | 19 ++++++++++--------- doc/sandboxed-compiles.md | 4 ++-- doc/the-doctor.md | 8 ++++---- doc/tls-proxy.md | 15 +++++++++++++-- lib/config-seed/overleaf.rc | 10 +++++----- lib/config-seed/variables.env | 122 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------------------------------------------------- lib/config-seed/version | 2 +- lib/default.rc | 4 ++-- lib/docker-compose.base.yml | 6 ++---- lib/docker-compose.sibling-containers.yml | 4 ++-- lib/docker-compose.vars-legacy.yml | 8 ++++++++ lib/docker-compose.vars.yml | 8 ++++++++ lib/shared-functions.sh | 67 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++---------- 24 files changed, 372 insertions(+), 165 deletions(-) create mode 100755 bin/rename-rc-vars create mode 100644 lib/docker-compose.vars-legacy.yml create mode 100644 lib/docker-compose.vars.yml Relaunching bin/upgrade after code update Skipping git update Rebranding from ShareLaTeX to Overleaf Found 3 lines with SHARELATEX_ in config/overleaf.rc Will create backup of config/overleaf.rc at config/__old-overleaf.rc.2024.02.26-17.10.14 Proceed with the renaming in config/overleaf.rc? (y/n): y Creating backup file config/__old-overleaf.rc.2024.02.26-17.10.14 Replacing 'SHARELATEX_' with 'OVERLEAF_' in config/overleaf.rc Updated 3 lines in config/overleaf.rc New docker image version available (5.0.0-RC) Current image version is '4.4.0' (from config/version) WARNING: this is a major version update, please check the Release Notes for breaking changes before proceeding: * https://github.com/overleaf/overleaf/wiki#release-notes Upgrade image? [y/n] y Upgrading config/version from 4.4.0 to 5.0.0-RC docker services are up, stop them first? Stop docker services? [y/n] y Stopping docker services [+] Stopping 1/1 ✔ Container mongo Stopped 1.1s At this point, we recommend backing up your data before proceeding !! WARNING: Only do this while the docker services are stopped!! Proceed with the upgrade? [y/n] y Backing up old version file to config/__old-version Over-writing config/version with 5.0.0-RC Rebranding from ShareLaTeX to Overleaf Found 45 lines with SHARELATEX_ in config/variables.env Will create backup of config/variables.env at config/__old-variables.env.2024.02.26-17.10.34 Proceed with the renaming in config/variables.env? (y/n): y Creating backup file config/__old-variables.env.2024.02.26-17.10.34 Replacing 'SHARELATEX_' with 'OVERLEAF_' in config/variables.env Updated 45 lines in config/variables.env Start docker services again? [y/n] y Starting docker services [+] Running 1/1 ✘ sharelatex Error Error response from daemon: manifest for quay.io/sharelatex/sharelatex-pro:5.0.0-RC not found: manifest unknown: manifest unknown (that error is expected) ```

das7pad commented 7 months ago

@briangough could you take another look at the commits I added today? Thanks!

das7pad commented 7 months ago

Rebased to fix the conflicts (bonus: I've also switched to 5.0.0-RC1 while fixing the version conflict).

das7pad commented 7 months ago

Fix for https://github.com/overleaf/internal/issues/17375 https://github.com/overleaf/toolkit/commit/f8d14ed8afb6ced21455203b238fa938d72932be

toolkit$ bin/up
Initiating Mongo replica set...
Rebranding from ShareLaTeX to Overleaf
  The 'TEXMFVAR' override is not needed since Server Pro/Overleaf CE version 3.2 (August 2022) and it conflicts with the rebranded paths.
  Please remove the following entry from your config/variables.env:

    TEXMFVAR=/var/lib/sharelatex/tmp/texmf-var

overleaf / toolkit