search

getporter / porter

Porter enables you to package your application artifact, client tools, configuration and deployment logic together as an installer that you can distribute, and install with a single command.
https://porter.sh
Apache License 2.0
1.23k stars 206 forks source link

How to diagnose Porter failure to migrate storage? #1501

Closed philliphoff closed 3 years ago

philliphoff commented 3 years ago

In a VS Codespace (a Linux VM), I get the below failure. Any idea how to go about diagnosing failures in the storage migration or understanding which schema, in particular, Porter is have trouble with? (I haven't come across much documentation of the schemas themselves.)

node ➜ /workspaces/webapi-nodejs-template-3/src/bundle (main ✗) $ porter --version
porter v0.34.4 (ccebf071)
node ➜ /workspaces/webapi-nodejs-template-3/src/bundle (main ✗) $ porter install --cred ./creds.json --parameter-set ./params.json
installing webapi-nodejs-3...
Error: could not save the pending action's status, the bundle was not executed: The schema of Porter's data is in an older format than supported by this version of Porter. 
Refer to https://porter.sh/storage-migrate for more information and instructions to back up your data. 
Once your data has been backed up, run the following command to perform the migration:

    porter storage migrate

node ➜ /workspaces/webapi-nodejs-template-3/src/bundle (main ✗) $ porter storage migrate

Saved migration logs to 
Error: Migration failed!
node ➜ /workspaces/webapi-nodejs-template-3/src/bundle (main ✗) $
carolynvs commented 3 years ago

You can use --debug --debug-plugins to see more information on what is being read out of storage.

Here's some more info on the schema and how we detect if we need to migrate:

  1. Look for schema.json at the root porter's storage (depending on the plugin used, this may be in ~/.porter/schema.json or it could be stored in Azure table storage for example).
  2. If it can't be found and PORTER_HOME is empty, then we assume you have a brand new installation. Porter initializes schema.json with default values.
  3. If PORTER_HOME is not empty, then we assume that you have an old version of Porter from before we started tracking storage schema. That triggers an error about requiring a migration.
  4. If we do find a schema.json, we compare the versions in that file with the versions supported by the current porter binary. If they don't match, that also triggers the error about requiring a migration.

Porter never migrates your data implicitly. In this case, you would run the requested migration comment, e.g. porter storage migrate and Porter migrates existing data and updates the schema.json to match.

FWIW, we haven't done anything that would trigger a migration since ~July 2020. Based on that and the error you are seeing when you try to run porter storage migrate, I would guess that porter is having trouble communicating with the storage backend or the data being returned from the plugin isn't in the expected structure. Hopefully the plugin debug logs will help isolate where the problem is.

philliphoff commented 3 years ago

@carolynvs Thanks, I'll poke around some more. This is a brand new VM with the latest Porter client tools. There is no schema.json in ~/.porter, nor is there a PORTER_HOME environment variable defined.

node ➜ /workspaces/webapi-nodejs-template-3/src/bundle (main ✗) $ porter install --debug --cred ./creds.json --parameter-set ./params.json
DEBUG name:    arm
DEBUG pkgDir: /home/node/.porter/mixins/arm
DEBUG file:     
DEBUG stdin:

/workspaces/webapi-nodejs-template-3/src/bundle/home/node/.porter/mixins/arm/arm version --output json --debug
DEBUG name:    aws
DEBUG pkgDir: /home/node/.porter/mixins/aws
DEBUG file:     
DEBUG stdin:

/workspaces/webapi-nodejs-template-3/src/bundle/home/node/.porter/mixins/aws/aws version --output json --debug
DEBUG name:    az
DEBUG pkgDir: /home/node/.porter/mixins/az
DEBUG file:     
DEBUG stdin:

/workspaces/webapi-nodejs-template-3/src/bundle/home/node/.porter/mixins/az/az version --output json --debug
DEBUG name:    exec
DEBUG pkgDir: /home/node/.porter/mixins/exec
DEBUG file:     
DEBUG stdin:

/workspaces/webapi-nodejs-template-3/src/bundle/home/node/.porter/mixins/exec/exec version --output json --debug
DEBUG name:    gcloud
DEBUG pkgDir: /home/node/.porter/mixins/gcloud
DEBUG file:     
DEBUG stdin:

/workspaces/webapi-nodejs-template-3/src/bundle/home/node/.porter/mixins/gcloud/gcloud version --output json --debug
DEBUG name:    helm
DEBUG pkgDir: /home/node/.porter/mixins/helm
DEBUG file:     
DEBUG stdin:

/workspaces/webapi-nodejs-template-3/src/bundle/home/node/.porter/mixins/helm/helm version --output json --debug
DEBUG name:    kubernetes
DEBUG pkgDir: /home/node/.porter/mixins/kubernetes
DEBUG file:     
DEBUG stdin:

/workspaces/webapi-nodejs-template-3/src/bundle/home/node/.porter/mixins/kubernetes/kubernetes version --output json --debug
DEBUG name:    terraform
DEBUG pkgDir: /home/node/.porter/mixins/terraform
DEBUG file:     
DEBUG stdin:

/workspaces/webapi-nodejs-template-3/src/bundle/home/node/.porter/mixins/terraform/terraform version --output json --debug
installing webapi-nodejs-3...
Error: could not save the pending action's status, the bundle was not executed: The schema of Porter's data is in an older format than supported by this version of Porter. 
Refer to https://porter.sh/storage-migrate for more information and instructions to back up your data. 
Once your data has been backed up, run the following command to perform the migration:

    porter storage migrate

node ➜ /workspaces/webapi-nodejs-template-3/src/bundle (main ✗) $ porter storage migrate --debug

Saved migration logs to 
Error: Migration failed!
node ➜ /workspaces/webapi-nodejs-template-3/src/bundle (main ✗) $
carolynvs commented 3 years ago

Okay okay so it's using the filesystem plugin. I noticed that your command didn't include --debug-plugins which is the other flag needed to see communications with the storage plugin.

philliphoff commented 3 years ago

@carolynvs I noticed that your command didn't include --debug-plugins which is the other flag needed to see communications with the storage plugin.

Ah, sorry about that, let me try again.

node ➜ /workspaces/webapi-nodejs-template-3/src/bundle (main ✗) $ porter install --debug --debug-plugins --cred ./creds.json --parameter-set ./params.json
DEBUG name:    arm
DEBUG pkgDir: /home/node/.porter/mixins/arm
DEBUG file:     
DEBUG stdin:

/workspaces/webapi-nodejs-template-3/src/bundle/home/node/.porter/mixins/arm/arm version --output json --debug
DEBUG name:    aws
DEBUG pkgDir: /home/node/.porter/mixins/aws
DEBUG file:     
DEBUG stdin:

/workspaces/webapi-nodejs-template-3/src/bundle/home/node/.porter/mixins/aws/aws version --output json --debug
DEBUG name:    az
DEBUG pkgDir: /home/node/.porter/mixins/az
DEBUG file:     
DEBUG stdin:

/workspaces/webapi-nodejs-template-3/src/bundle/home/node/.porter/mixins/az/az version --output json --debug
DEBUG name:    exec
DEBUG pkgDir: /home/node/.porter/mixins/exec
DEBUG file:     
DEBUG stdin:

/workspaces/webapi-nodejs-template-3/src/bundle/home/node/.porter/mixins/exec/exec version --output json --debug
DEBUG name:    gcloud
DEBUG pkgDir: /home/node/.porter/mixins/gcloud
DEBUG file:     
DEBUG stdin:

/workspaces/webapi-nodejs-template-3/src/bundle/home/node/.porter/mixins/gcloud/gcloud version --output json --debug
DEBUG name:    helm
DEBUG pkgDir: /home/node/.porter/mixins/helm
DEBUG file:     
DEBUG stdin:

/workspaces/webapi-nodejs-template-3/src/bundle/home/node/.porter/mixins/helm/helm version --output json --debug
DEBUG name:    kubernetes
DEBUG pkgDir: /home/node/.porter/mixins/kubernetes
DEBUG file:     
DEBUG stdin:

/workspaces/webapi-nodejs-template-3/src/bundle/home/node/.porter/mixins/kubernetes/kubernetes version --output json --debug
DEBUG name:    terraform
DEBUG pkgDir: /home/node/.porter/mixins/terraform
DEBUG file:     
DEBUG stdin:

/workspaces/webapi-nodejs-template-3/src/bundle/home/node/.porter/mixins/terraform/terraform version --output json --debug
Resolved secrets plugin to secrets.porter.host
/home/node/.porter/porter plugin run secrets.porter.host
Resolved secrets plugin to secrets.porter.host
/home/node/.porter/porter plugin run secrets.porter.host
Resolved secrets plugin to secrets.porter.host
/home/node/.porter/porter plugin run secrets.porter.host
Resolved secrets plugin to secrets.porter.host
/home/node/.porter/porter plugin run secrets.porter.host
error closing client during Kill
plugin failed to exit gracefully
Resolved secrets plugin to secrets.porter.host
/home/node/.porter/porter plugin run secrets.porter.host
Resolved secrets plugin to secrets.porter.host
/home/node/.porter/porter plugin run secrets.porter.host
Resolved secrets plugin to secrets.porter.host
/home/node/.porter/porter plugin run secrets.porter.host
Resolved secrets plugin to secrets.porter.host
/home/node/.porter/porter plugin run secrets.porter.host
installing webapi-nodejs-3...
Resolved storage plugin to storage.porter.filesystem
/home/node/.porter/porter plugin run storage.porter.filesystem
PORTER HOME: /home/node/.porter
Resolved secrets plugin to secrets.porter.host
/home/node/.porter/porter plugin run secrets.porter.host
Resolved secrets plugin to secrets.porter.host
/home/node/.porter/porter plugin run secrets.porter.host
Resolved secrets plugin to secrets.porter.host
/home/node/.porter/porter plugin run secrets.porter.host
Resolved secrets plugin to secrets.porter.host
/home/node/.porter/porter plugin run secrets.porter.host
Resolved storage plugin to storage.porter.filesystem
/home/node/.porter/porter plugin run storage.porter.filesystem
PORTER HOME: /home/node/.porter
Error: could not save the pending action's status, the bundle was not executed: The schema of Porter's data is in an older format than supported by this version of Porter. 
Refer to https://porter.sh/storage-migrate for more information and instructions to back up your data. 
Once your data has been backed up, run the following command to perform the migration:

    porter storage migrate
carolynvs commented 3 years ago

Weird! A couple thoughts:

  1. You should also be able to get this migration check to trigger by running porter list --debug --debug-plugins. It will fail faster and with less extraneous output I think.
  2. Do you have write access to /home/node/.porter?
  3. Can you run tree -la /home/node/.porter (if you don't have tree, I think ls -laR /home/node/.porter)
philliphoff commented 3 years ago

@carolynvs

  1. Do you have write access to /home/node/.porter?
  2. Can you run tree -la /home/node/.porter (if you don't have tree, I think ls -laR /home/node/.porter)
node ➜ /workspaces/webapi-nodejs-template-3/src/bundle (main ✗) $ ls -laR /home/node/.porter
/home/node/.porter:
total 56836
drwxr-xr-x  5 root root     4096 Mar 16 18:15 .
drwxr-xr-x  1 node node     4096 Mar 16 20:39 ..
drwxr-xr-x 10 root root     4096 Mar 16 18:15 mixins
drwxr-xr-x  3 root root     4096 Mar 16 18:15 plugins
-rwxr-xr-x  1 root root 58113123 Mar 16 18:15 porter
drwxr-xr-x  2 root root     4096 Mar 16 18:15 runtimes

Ah, that looks a bit suspicious. Even moreso when I go back to the Dockerfile:

## Porter
RUN curl https://cdn.porter.sh/latest/install-linux.sh | bash
RUN sudo mv /root/.porter /home/node/.porter
ENV PATH "$PATH:/home/node/.porter"

I'll have to look into why Porter is installed that way.

Thanks for the pointers!

carolynvs commented 3 years ago

Oh, yeah looks like the file permissions aren't for the correct user anymore? Not sure where the dockerfile came from, let me know if this was something we put out!

philliphoff commented 3 years ago

The Dockerfile came from our team. I think the mv was added because the installation happens as root during the build but the ultimate user is node. One question: is Porter intended to be installed on a machine for all users or for individual users?

carolynvs commented 3 years ago

Right now it's designed to be installed for each user. Then if a team wants to manage installations of bundles together, they would use a storage plugin so that Porter isn't writing to ~/.porter.

philliphoff commented 3 years ago

Ok, I've updated our Dockerfile to explicitly install Porter as the user being used within the container, and things work more smoothly now. Thanks for the help!