search

f5devcentral / f5-journeys

F5 Journeys Migration Utility - migrate or upgrade BIG-IP into new F5 platforms
Apache License 2.0
77 stars 17 forks source link
as3 big-ip diagnostics migration per-app rseries ucs upgrade velos

readme

JOURNEYS - BIG-IP upgrade and migration utility

Contents:

Description

JOURNEYS is an application designed to assist F5 Customers with migrating a BIG-IP configuration to a new F5 device and enable new ways of migrating.

Supported journeys:

Read also the Journeys App solution overview on the F5.com page.

Journey: Full Config migration

Supported features:

Full config BIG-IP migrations are supported for software paths according to the following matrix:

DEST
X 11.x 12.x 13.x 14.x 15.x 16.x 17.x
<11.5 X* X X X^ X^
12.x X X X X^
SRC 13.x X X X
14.x X X X
15.x X X X
16.x X** X
17.x X

X^ - an exception compared to the supported upgrade paths listed in the official document K13845 (upgrade allowed only if the source configuration is upgraded to the latest available maintenance release).

X* - while an upgrade to a 11.x destination system is technically supported on BIG-IP, JOURNEYS does not support deployments to systems runnig these versions due to a lack of platform-migrate ucs load option (introduced in 12.1.3).

X** - not supported on rSeries or VELOS platforms.

WARNING: Migrating Application Services using keys stored in FIPS cards is not supported at the moment, unless the user can restore the FIPS keys with the original (non-fips) ones on the destination platform.

Known parity gaps

JOURNEYS finds the following configuration elements in the source configuration, which are no longer supported by the BIG-IP running on the destination platform. For every incompatible element found, there will be one or more automatic possible solutions provided.

Common issues

Velos and rSeries specific issues

JOURNEYS does not support feature parity gaps that:

BIG-IP Prerequisites

Mandatory steps before running Full Config migration in JOURNEYS:

  1. Master key transfer - to allow handling encrypted objects, before running JOURNEYS, you need to set a device master key password on both Source and Destination Systems. There are two ways to do this:

    1. Copy the Source System master key with f5mku and re-key master key on the Destination System:

      Important: Whenever possible, use the documented tmsh commands for master and unit key manipulation. Only use the f5mku command with assistance from F5 Support when no tmsh commands exist to perform the task you want.

      • Obtain the master key from the Source System by entering the following commands: 
        f5mku -K

        and copy the output. The command output appears similar to the following example:

        oruIVCHfmVBnwGaSR/+MAA==
      • Install the master-key that you copied in the previous step to the Destination System using the following syntax: 
        f5mku -r <key_value>

    2. Set master-key password before saving the source UCS file

      • Set the device master key password on the Source System by entering the following commands (remember this password because you'll need to use the same password on the destination device)

        tmsh modify sys crypto master-key prompt-for-password
tmsh save sys config

      • Set the master key password on the Destination System by entering the following commands, using the password remembered in the previous step:

        tmsh modify sys crypto master-key prompt-for-password
tmsh save sys config

    For more details, please refer to:

  2. SSH public keys migration

    • SSH public keys for passwordless authentication may stop work after UCS migration, since the UCS file may not contain SSH public keys for users.
    • If the version is affected by the problem, then:
      • all key files have to be migrated manually from the Source System to the Destination System
      • /etc/ssh directory has to be added to the UCS backup configuration of the Source System
    • For more details on how to manually migrate SSH keys and verify if your version is affected by the problem, please read:

  3. Destination System preparation for JOURNEYS

    1. Destination BIG-IP should be in Active state.
    2. If migrating from BIG-IP 14.0.x or lower, ensure that mgmt-dhcp value in sys global-settings on the Destination System is set to either disabled or enabled. Any other values - namely dhcpv4 and dhcpv6, available starting at 14.1.0 - will cause an error during configuration loading.
    3. VLANs, trunks and interfaces should already be configured on vCMP, rSeries and VELOS systems. For more details, please refer to: Platform-migrate option overview: K82540512.
      For VELOS or rSeries tenant configuration, find more information on F5OS CLI.
      VLANs and trunks creation example: 
      config
vlans vlan 1000 config name Vlan1000 vlan-id 1000
vlans vlan 1001 config name Vlan1000 vlan-id 1001
vlans vlan 1002 config name Vlan1000 vlan-id 1002
top
interfaces interface 1.0 ethernet switched-vlan config trunk-vlans 1000
interfaces interface 1.0 ethernet switched-vlan config trunk-vlans 1001
interfaces interface 1.0 ethernet switched-vlan config trunk-vlans 1002
commit
end

  4. FIPS/NetHSM keys migration

    • Keys stored in FIPS/NetHSM will not be migrated, since the UCS file does not contain keys of these types.
    • If FIPS/NetHSM keys are in use by the platform, then:
      • all key files have to be migrated manually from the Source System to the Destination System
    • For more details on how to manually migrate keys stored in FIPS/NetHSM, please read:

BIG-IP account prerequisites

To ensure all JOURNEYS features work properly, an account with Administrator role and advanced shell (bash) access is required on both source and target hosts. It can be root or any other account. For auditing purposes, a separate account for migration might be desired.

IMPORTANT: Due to the above, certain features of JOURNEYS - specifically the ones requiring ssh access to the machine - are not available on BIG-IPs running in the Appliance mode. The user will be required to perform manual variants of these steps instead.

Journey: Application Service Migration

Supported features:

Configuration object grouping

  1. Virtual - smallest object recognized by JOURNEYS. It includes a single ltm virtual-server (represented by a Service in AS3) object and any others referenced by it directly or indirectly - monitors, pools, profiles, etc.

  2. Application - A group of AS3 objects, including the virtual mentioned above. Logic for initial grouping of virtuals into apps can be manually set via preferences in GUI before loading the ucs. Represented by folders on the BIG-IP level.

  3. Tenant - A group of AS3 applications. By default JOURNEYS creates one tenant per application. Common is a special reserved name, representing objects shared between tenants. Represented by partitions on the BIG-IP level.

Application conversion status

Each virtual separately gets assigned a status based on the f5-automation-config-converter response. Possible statuses are as follows:

Note: If one or more of your apps have a red or black status, you may attempt to use a newer version of f5-acc-config-converter by editing the image version inside the docker-compose.yml file. Otherwise, please open an issue on f5devcentral and include configuration contents from the problematic app.

Additional notes

Deployable objects

Not all required configuration files can be included inside the resulting AS3 json, and need to be installed on the Destination System prior to sending the declaration itself.

If deploying manually, JOURNEYS will prepare a package containing all of the required files alongside with a list of commands to perform on the Destination System.

If deploying via JOURNEYS, the application will install the files automatically.

Known issues

Configuration Migration Considerations

BIG-IP device swap

To minimize downtime, F5 recommends deploying the new hardware alongside existing BIG-IP deployment.

F5 recommends the following procedure for moving production traffic to a new device:

  1. Deploy the target device, trying to keep physical connections as close as possible to the old BIG-IP (respective interfaces assigned to the same physical networks)
  2. Remove interfaces to existing VLANs on the new hardware (this will impact all tenants on VELOS/rSeries/vCMP guests).
    There are three options to do this:
    1. Disabling interfaces.
    2. Physically unplugging the network cables.
    3. Disabling the port on the switch connected to the destination platform.
  3. Deploy JOURNEYS-generated config to a new BIG-IP. Note that some validators like LTM module comparison are expected to fail as Virtual Servers will be down.
  4. If the configuration was deployed successfully, review the system status. If the status is "REBOOT REQUIRED", perform the reboot before shutting down interfaces on the old BIG-IP system.
  5. If you use BIG-IQ, refer to the article K15938 to discover your new BIG-IP device.
  6. Shutdown interfaces on the old BIG-IP system (this will impact all source BIG-IPs).
    There are three options to do this:
    1. Disabling interfaces.
    2. Physically unplugging the network cables.
    3. Disabling the port on the switch connected to the old BIG-IP system.
  7. Re-add interfaces to existing VLANs on the new hardware (this will impact all tenants on VELOS/rSeries/vCMP guests).
    There are three options to do this:
    1. Re-enable interfaces on the destination platform.
    2. Physically unplug the network cables on the source platform, plug the network cables on the destination platform.
    3. Enable the port on the switch connected to the destination platform.

SPDAG/VlanGroup mitigation

If SPDAG or VlanGroup removal mitigation is applied, and a conflicted object is configured on a Virtual Server, JOURNEYS will remove all VLANs assigned for that particular Virtual Server - not only the conflicted one. This is done to ensure that JOURNEYS does not produce an invalid configuration (Virtual Servers cannot share identifiers, as they need to be unique).

Rebuilding device trust:

During migration JOURNEYS will reset the device trust.
F5 reccomends one of the following procedures to rebuild the device trust:

Restoring backup UCS in case of migration failure

Deployment progress can be tracked on the Summary page. If UCS load fails, it is strongly advised to manually load the backup UCS archive created during the deployment process before any other action is taken on the Destination System.

  1. Log to the Destination System: 
      ssh username@<destination-system-ip>
  2. Use the command to load the backup UCS: 
      tmsh load sys ucs <backup_ucs_name*>

    *The backup UCS archive name is displayed on the Summary page in the Detailed Results in a log entry that should look as follows:

      Create backup of the Destination System: /var/local/ucs/journey_ucs_backup_20211012-103937.ucs

JOURNEYS Setup Requirements

System requirements

JOURNEYS Installation

Fetching JOURNEYS

git clone https://github.com/f5devcentral/f5-journeys.git
cd f5-journeys

Airgapped installation using a bundle

  1. Download the bundle and the checksum file from Releases or from MyF5 using a device with internet access.

  2. Check the checksum file from the downloaded bundle, see examples for Linux:

    sha512sum -c f5-journeys-bundle-v4.1.0.tgz.sha512

or:

md5sum -c f5-journeys-bundle-v4.1.0.tgz.md5

Note: Do not use the application if the message digest does not match the downloaded file.

  1. (optional): Upload the file to target device using your preferred method.

  2. Unpack the archive content, for example:

    tar zxvf f5-journeys-bundle-v4.1.0.tgz
cd f5-journeys

  3. Import containers into Docker or other container runtime using following script:

./install.sh
  1. Continue directly to steps 1-2 and 6-7 in Preparing the environment.

Preparing the environment

  1. Create a directory for all of JOURNEYS operations - modifying configs, logging, etc.

    You can use any directory in place of /tmp/journeys. 

    mkdir /tmp/journeys

    Note: keep in mind that the default migration directory is placed in the /tmp directory, which is cleaned up upon a system reboot. If longer session persistence is required, please modify this directory path via the .env file to a non-temporary one.

  2. Prepare an environment file for docker-compose

    cp sample.env .env

    If you are using a different working directory than the one shown in the point above, WORKING_DIRECTORY variable has to be updated in the .env file. Its value should be an absolute path pointing to your custom directory.

  3. Fetch services included in the docker-compose configuration file

    docker compose pull

  4. Print sha digest of downloaded images

    docker image ls f5devcentral/f5-bigip-journeys-app --digests

  5. Examine if image listed in docker-compose journeys/celery-worker has exactly the same digest that can be found on docker hub repository with the same tag

    https://hub.docker.com/r/f5devcentral/f5-bigip-journeys-app/tags?page=1&ordering=last_updated

    Note: do not use the application if digests does not match.

  6. Start services included in the docker-compose configuration file

    docker compose up -d

    Note: services included in the default docker-compose.yaml file do not allow usage of the perapp functionalities. To use them, please refer to perapp section.

    Note: for security reasons, not all the ports in docker-compose.yaml are open to the public, such as Redis(6379). To debug Redis, please modify the docker-compose.yaml file to add ports: exposing session and run the up command again.

  7. Open your web browser, navigate to:

    https://localhost:8443

    and accept self-signed cert to run the application. The certificate itself can be verified by comparing it with the one logged by the main JOURNEYS container.

    docker logs f5-journeys_journeys-1 2>&1 | grep 'BEGIN CERTIFICATE'

    if the above command returns that no container has been found look for the name of deployment container like so

    docker ps

    and replace the journeys container name with the one you have found with the docker ps command

    docker logs <your journeys container name> 2>&1 | grep 'BEGIN CERTIFICATE'

JOURNEYS Update

To update Journeys to the latest version, run the following commands.

  1. Clean up the old working directory. Sometimes leftover data might not be compatible with new Journeys version. 
    rm -rf /tmp/journeys  # or whichever folder you specified in the .env file
mkdir /tmp/journeys
  2. Ensure that your current .env file contains all keys present in the sample.env file. If not, copy/enter them.
  3. Bring down the old containers, pull the newest changes and start the services again. 
    docker compose down
git pull
docker compose up -d

Older images may then be viewed and removed using the following commands.

docker images | grep journeys
docker rmi <image_name>:<version>

JOURNEYS API

An API is exposed by the main journeys container (by default on port 8443). Available endpoints are shown in the openapi-schema.yaml file, which can also be imported in a tool like Postman or Swagger UI for easier exploring and/or usage.

Known API issues

Feature details

Diagnostics

Upon deployment of UCS to the Destination System, user will be prompted to select several diagnose methods. These collect information relevant to your system condition and compare its state and configuration with the Source BIG-IP System.

Please keep in mind that only a few of the diagnostics methods can be run without providing a Source System. If the UCS was provided manually and no such system was provided, unavailable options will be locked out.

IMPORTANT: User will always be prompted for source and destination credentials right before deployment even if Source System is offline. Destination credentials are valid before loading UCS archive on the target system. Once the UCS is loaded, source credentials (stored at UCS archive provided by a user or downloaded from the Source System) are required to log in.

Details of the diagnostic checks can be evaluated in a resulting pdf report.

Diagnose methods * ### MCP status check Area:| error detection -----|----- Checks if values of returned fields are correct. This method uses `tmsh show sys mcp-state field-fmt` that can be executed manually. * ### TMM status Area:| resource management -----|----- Function logs status of TMM. Requires manual evaluation. * ### Prompt state Area:| error detection -----|----- Checks if prompt state is in active mode. * ### Core dumps detection Area:| error detection -----|----- Checks if diagnostic core dumps were created. * ### Database Comparison Area:| config migration -----|----- Compares two system DBs getting them from iControl endpoint for sys db. Requires manual evaluation. * ### Memory footprint Comparison Area:| information, resource management -----|----- Compares information from `tmsh show sys provision` for both systems. Requires manual evaluation. * ### Version Comparison Area:| information -----|----- Compares information from `tmsh show sys version` for both systems. Requires manual evaluation. * ### Local Traffic Manager (LTM) module comparison checks Area:| config migration, resource management -----|----- Check lists of all defined LTM nodes and Virtual Servers configured in the new system. If both devices are on-line, it will check conformance of both configuration and resource availability. Requires manual evaluation. * ### Log Watcher check Area:| error detection -----|----- Log watcher checks the difference in log files before and after UCS deployment. This check looks for "ERR" and "CRIT" phrases (case-insensitive) that might appear in logs during the deployment. Current scope of log file lookup: - /var/log/ltm" - /var/log/apm" - /var/log/gtm" - /var/log/tmm" - /var/log/liveinstall.log - /var/log/asm" - /var/log/ts/bd.log - /var/log/ts/asm_config_server.log - /var/log/ts/pabnagd.log - /var/log/ts/db_upgrade.log - /var/log/daemon.log - /var/log/kern.log - /var/log/messages The log watcher output only includes these files, which appear on your system and are provisioned. Sample output: ```json Log watcher output: { "/var/log/ltm": [], } ``` Empty list as a value means no lines containing phrases "ERR" and "CRIT" were found. However, there may be information about potential problems in logs. Therefore, this check requires manual evaluation. * ### Ciphersuites check Area:| information -----|----- Compares information about ciphersuites for both systems. Requires manual evaluation. * ### Cron entries check Area:| error detection -----|----- Compares cron data between Source and Destination Systems, checking for any unexpected changes. * ### Platform migrate ignored objects Area:| information -----|----- Lists objects ignored due to the usage of `platform migrate` ucs load option. Requires manual evaluation. * ### Services check Area:| error detection -----|----- Verifies whether any services on the Destination System are stuck in a restart loop. If they are, attempt to fix the issue by rebooting the device. > WARNING: this check can include a reboot. Using it might significantly increase validation runtime, and can cause unexpected changes on the target device.

Load validation

During the migration process, user will be provided with an option to validate if the configuration being migrated will load on the provided Destination System. This operation utilizes the BIG-IP built in tmsh sys config verify functionality to check whether the Destination BIG-IP finds any errors in the provided files. This module temporarily modifies Destination configuration files for testing purposes. After moving on from this step, the Destination System will be reverted back to its original state.

WARNING: Due to how tmsh sys config verify works, the load validation module has to temporarily heavily modify configuration files on the provided Destination System. For this reason, to avoid any potential loss of data, it is not recommended to use this feature with a device already containing any non-basic configuration.

WARNING: Please make sure the Destination device is licensed and provisioned with modules corresponding to the UCS configuration being migrated.

WARNING: Load validation can produce false positive output if the configuration contains any parts subject to fix-up scripts built into the BIG-IP ucs load process. These fix-ups cannot be automatically incorporated in the validation module.

WARNING: If the configuration to be validated requires specific values in BigDB.dat file, they must be configured on a destination device before performing the validation, otherwise "load validation" may fail.

Load validation functional details

Since the verify command requires an unpacked and ready to load configuration on a live system, this module needs to perform a few preliminary steps before actual validation. The whole process looks as follows:

  1. Modify the configuration files:

    • Remove several objects from the net module: interface, stp, vlan, fdb vlan, trunk
    • Remove any references to the removed objects
    • Disable user session limit

  2. Generate a temporary UCS with the current configuration and upload it to the provided device

  3. Generate a backup UCS

  4. Extract a part of the UCS file on the provided host, including but not limited to:

    • Configuration files
    • Certificates and keys
    • Database files

  5. If the migration includes an AS3 file, check and potentially install the appsvcs package

  6. If the configuration to be validated requires specific values in BigDB.dat, configure them on a destination device. Example:

    • Destination device (statemirror.secure value "disable" is configured): 
      [root@localhost:Active:Standalone] config # tmsh list sys db statemirror.secure all-properties
sys db statemirror.secure {
default-value "disable"
scf-config "true"
value "disable"
value-range "disable enable"
}
[root@localhost:Active:Standalone] config #
    • Load validation error (statemirror.secure value "enable" is required): 
      01071958:3: Error configuring Virtual Server (/Common/virtual_server_HTTPS). Connection mirroring requires db variable statemirror.secure to be enabled.
Unexpected Error: Validating configuration process failed.
    • Setting required value of statemirror.secure on the destination device: 
      tmsh modify sys db statemirror.secure value enable

  7. Call the tmsh load sys config verify partitions all command and check its output. Additionally, verify whether mcpd is in correct state after this operation.

  8. If the migration includes an AS3 file, send a request with modified declaration having its action set as dry-run and verify the output

After the validation is complete, the test host is then restored to the original state:

  1. Restore the backup ucs
  2. Remove any files that were extracted from the verified UCS file, but were not present originally on the system
  3. Remove the test UCS

Contributing

See here

Bug reporting and Feature Requests

See here