How to migrate data from version 0.6.9 to system 0.7.0

[401557122]

Self Checks

• [X] This is only for bug report, if you would like to ask a question, please head to Discussions.
• [X] I have searched for existing issues search for existing issues, including closed ones.
• [X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• [X] Please do not modify this template :) and fill in all the required fields.

Dify version

0.7.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

How to migrate data from version 0.6.9 to system 0.7.0

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

[dosubot[bot]]

To migrate data from version 0.6.9 to version 0.7.0 in a self-hosted Dify setup using Docker, follow these documented steps:

1. Enter the docker directory of the Dify source code:

   cd dify/docker

2. Bring down the current Docker containers:

   docker compose down

3. Pull the latest changes from the main branch:

   git pull origin main

4. Pull the latest Docker images:

   docker compose pull

5. Bring up the Docker containers with the new version:

   docker compose up -d

These steps will handle the upgrade process, including any necessary data migrations ^[1].

To continue talking to Dosu, mention @dosu.

[401557122]

I need to migrate data, but the data table structure is different in these two versions. I copied the volume directly and encountered an error when starting @dosu

[dosubot[bot]]

To migrate data when the data table structure is different between version 0.6.9 and version 0.7.0 in a self-hosted Dify setup using Docker, follow these steps:

1. Review Changes: Familiarize yourself with the new .env configuration and Docker Compose setup.

2. Transfer Customizations:

   • If you have customized configurations such as docker-compose.yaml, ssrf_proxy/squid.conf, or nginx/conf.d/default.conf, reflect these changes in the .env file you create.

3. Data Backup:

   • Back up all data content in the dify/docker/volumes directory if deployed in Docker Compose mode.

4. Update and Migrate:

   • Pull the latest code or images:

     git pull origin main
     docker compose pull

   • Bring down the current Docker containers:

     docker compose down

   • Bring up the Docker containers with the new version:

     docker compose up -d

5. Database Migration:

   • Enter the API directory and execute the following command to migrate the database structure to the latest version:

     flask db upgrade

The migration script for version 0.7.0 includes creating new tables and indices, as well as dropping obsolete tables and indices ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[401557122]

It seems that there are some account password information stored in my old version of Volume. I found that when I migrated the data, if I modify the account password information in the configuration of the new version, it will appear that these information are inconsistent in the environment variables and data volume @dosu.

[dosubot[bot]]

To handle inconsistencies between account password information in the environment variables and data volume during a migration, follow these steps:

1. Backup Data: Ensure you have a backup of your current data volume.

2. Review Environment Variables: Verify the environment variables related to account passwords in your .env file. These include:

   • INIT_PASSWORD
   • DB_PASSWORD
   • REDIS_PASSWORD
   • CELERY_BROKER_URL
   • QDRANT_API_KEY
   • MILVUS_PASSWORD
   • CHROMA_AUTH_CREDENTIALS
   • ORACLE_PWD

3. Update Environment Variables: Ensure that the values in your .env file match the expected values for the new version. Update them if necessary.

4. Migrate Data:

   • Bring down the current Docker containers:

     docker compose down

   • Pull the latest code or images:

     git pull origin main
     docker compose pull

   • Bring up the Docker containers with the new version:

     docker compose up -d

5. Database Migration:

   • Enter the API directory and execute the following command to migrate the database structure to the latest version:

     flask db upgrade

6. Update Account Passwords: If there are inconsistencies, you may need to manually update the account passwords in the database to match the new environment variables. Use the update_account_password method from the AccountService class to ensure passwords are hashed and salted correctly.

By following these steps, you can ensure that the account password information is consistent between the environment variables and the data volume ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[401557122]

I changed the account password in env, how to change it in volume @dosu.

[dosubot[bot]]

To change the account password in the volume after updating it in the environment variables, you need to update the password directly in the database. You can use the reset-password command provided in the Dify repository. Here are the steps:

1. Enter the Docker container:

   docker exec -it <container_name> /bin/bash

2. Run the reset-password command:

   flask reset-password --email <your_email> --new-password <new_password> --password-confirm <new_password>

This command will update the account password in the database to match the new password specified in the environment variables. Make sure to replace <container_name>, <your_email>, and <new_password> with the appropriate values.

Here is the relevant code for the reset-password command:

@click.command("reset-password", help="Reset the account password.")
@click.option("--email", prompt=True, help="The email address of the account whose password you need to reset")
@click.option("--new-password", prompt=True, help="the new password.")
@click.option("--password-confirm", prompt=True, help="the new password confirm.")
def reset_password(email, new_password, password_confirm):
    """
    Reset password of owner account
    Only available in SELF_HOSTED mode
    """
    if str(new_password).strip() != str(password_confirm).strip():
        click.echo(click.style("sorry. The two passwords do not match.", fg="red"))
        return

    account = db.session.query(Account).filter(Account.email == email).one_or_none()

    if not account:
        click.echo(click.style("sorry. the account: [{}] not exist .".format(email), fg="red"))
        return

    try:
        valid_password(new_password)
    except:
        click.echo(click.style("sorry. The passwords must match {} ".format(password_pattern), fg="red"))
        return

    # generate password salt
    salt = secrets.token_bytes(16)
    base64_salt = base64.b64encode(salt).decode()

    # encrypt password with salt
    password_hashed = hash_password(new_password, salt)
    base64_password_hashed = base64.b64encode(password_hashed).decode()
    account.password = base64_password_hashed
    account.password_salt = base64_salt
    db.session.commit()
    click.echo(click.style("Congratulations! Password has been reset.", fg="green"))

This will ensure that the password is updated correctly in the database ^[1].

To continue talking to Dosu, mention @dosu.

[401557122]

Including the account password for Postgres, the volume I migrated seems to have a password, but the new version of env wants to change it @dosu.

[dosubot[bot]]

To update the Postgres account password in the volume after changing it in the environment variables, follow these steps:

1. Update the POSTGRES_PASSWORD Environment Variable: Update the POSTGRES_PASSWORD environment variable in your docker-compose.yaml file or in the environment variables file (e.g., .env).

   Example in the .env file:

   POSTGRES_PASSWORD=newpassword123

2. Restart the db Service: Restart the db service to apply the new password.

   Example docker-compose.yaml:

   services:
    db:
      image: postgres:15-alpine
      restart: always
      env_file:
        - ./middleware.env
      volumes:
        - ./volumes/db/data:/var/lib/postgresql/data
      healthcheck:
        test: [ "CMD", "pg_isready" ]
        interval: 1s
        timeout: 3s
        retries: 30

   Restart the service:

   docker compose down
   docker compose up -d db

3. Manually Update the Postgres Password in the Database: Enter the Postgres container and update the password directly in the database.

   docker exec -it <postgres_container_name> psql -U postgres

   Once inside the Postgres shell, run:

   ALTER USER postgres WITH PASSWORD 'newpassword123';

   Replace <postgres_container_name> with the actual name of your Postgres container and newpassword123 with your new password.

By following these steps, you ensure that the Postgres password is updated both in the environment variables and directly within the database ^{[1][2][3][4][5]}.

To continue talking to Dosu, mention @dosu.

[crazywoola]

Please use Discussion for asking help. This is a template for issue report.