Refactored export script

gerald1248 commented 6 years ago

lists containing multiple objects are valid (not stacked) JSON
like the original, they can be restored via oc create -f
backup continues when a resource type is not recognised
backup continues when permission to access an object is refused (see issues #1039, #1043)
deletion of empty files is no longer necessary
the two exceptions (dc and svc), which create a separate file for each object, are unchanged

What does this PR do?

The current project_export.sh produces JSON files that can't be parsed by all processors and so may require work before they can be restored. (jq is fine but python -m json.tool complains about "Extra data".) This version produces List objects instead.

The commit also ensures the export handles resource types not known to the master (the user may be running an older OpenShift version) and types which the user/service account can't access (e.g. Secrets) gracefully. With this change, the script supports older OpenShift versions and non-admin users.

How should this be manually tested?

Log in to any OpenShift cluster and create a backup of an existing project (e.g. myproject):

$ ./project_export.sh myproject

Is there a relevant Issue open for this?

https://github.com/openshift/openshift-ansible-contrib/issues/1039 https://github.com/openshift/openshift-ansible-contrib/issues/1043 https://github.com/openshift/openshift-ansible-contrib/issues/1072

Who would you like to review this?

cc: @e-minguez

e-minguez commented 6 years ago

I've just tested it with latest minishift and:

oc new-project test001
oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-ex.git
./project_export.sh test001

And it fails with:

...
Exporting deploymentconfigs to test001/dc_*.json
Patching DC...
Exporting 'bc' resources to test001/bcs.json
jq: error (at <stdin>:1): Cannot iterate over null (null)

Debug output (sh -x):

++ jq '.items | length > 0'
++ echo '{' '"apiVersion":' '"v1",' '"items":' '[' '{' '"apiVersion":' '"build.openshift.io/v1",' '"kind":' '"BuildConfig",' '"metadata":' '{' '"annotations":' '{' '"openshift.io/generated-by":' '"OpenShiftNewApp"' '},' '"creationTimestamp":' '"2018-08-27T12:14:06Z",' '"labels":' '{' '"app":' '"ruby-ex"' '},' '"name":' '"ruby-ex",' '"namespace":' '"test001",' '"resourceVersion":' '"5429",' '"selfLink":' '"/apis/build.openshift.io/v1/namespaces/test001/buildconfigs/ruby-ex",' '"uid":' '"b1c2ab85-a9f2-11e8-8ed4-525400725045"' '},' '"spec":' '{' '"failedBuildsHistoryLimit":' 5, '"nodeSelector":' null, '"output":' '{' '"to":' '{' '"kind":' '"ImageStreamTag",' '"name":' '"ruby-ex:latest"' '}' '},' '"postCommit":' '{},' '"resources":' '{},' '"runPolicy":' '"Serial",' '"source":' '{' '"git":' '{' '"uri":' '"https://github.com/openshift/ruby-ex.git"' '},' '"type":' '"Git"' '},' '"strategy":' '{' '"sourceStrategy":' '{' '"from":' '{' '"kind":' '"ImageStreamTag",' '"name":' '"ruby-22-centos7:latest"' '}' '},' '"type":' '"Source"' '},' '"successfulBuildsHistoryLimit":' 5, '"triggers":' '[' '{' '"github":' '{' '"secret":' '"e8Z9PWAmgWTl5ahU8Ek6"' '},' '"type":' '"GitHub"' '},' '{' '"generic":' '{' '"secret":' '"qq7bCqRTNJKY9Kt9IKPt"' '},' '"type":' '"Generic"' '},' '{' '"type":' '"ConfigChange"' '},' '{' '"imageChange":' '{' '"lastTriggeredImageID":' '"centos/ruby-22-centos7@sha256:a18c8706118a5c4c9f1adf045024d2abf06ba632b5674b23421019ee4d3edcae"' '},' '"type":' '"ImageChange"' '}' ']' '},' '"status":' '{' '"lastVersion":' 1 '}' '}' '],' '"kind":' '"List",' '"metadata":' '{' '"resourceVersion":' '"",' '"selfLink":' '""' '}' '}'
+ '[' true == false ']'
+ echo '{' '"apiVersion":' '"v1",' '"items":' '[' '{' '"apiVersion":' '"build.openshift.io/v1",' '"kind":' '"BuildConfig",' '"metadata":' '{' '"annotations":' '{' '"openshift.io/generated-by":' '"OpenShiftNewApp"' '},' '"creationTimestamp":' '"2018-08-27T12:14:06Z",' '"labels":' '{' '"app":' '"ruby-ex"' '},' '"name":' '"ruby-ex",' '"namespace":' '"test001",' '"resourceVersion":' '"5429",' '"selfLink":' '"/apis/build.openshift.io/v1/namespaces/test001/buildconfigs/ruby-ex",' '"uid":' '"b1c2ab85-a9f2-11e8-8ed4-525400725045"' '},' '"spec":' '{' '"failedBuildsHistoryLimit":' 5, '"nodeSelector":' null, '"output":' '{' '"to":' '{' '"kind":' '"ImageStreamTag",' '"name":' '"ruby-ex:latest"' '}' '},' '"postCommit":' '{},' '"resources":' '{},' '"runPolicy":' '"Serial",' '"source":' '{' '"git":' '{' '"uri":' '"https://github.com/openshift/ruby-ex.git"' '},' '"type":' '"Git"' '},' '"strategy":' '{' '"sourceStrategy":' '{' '"from":' '{' '"kind":' '"ImageStreamTag",' '"name":' '"ruby-22-centos7:latest"' '}' '},' '"type":' '"Source"' '},' '"successfulBuildsHistoryLimit":' 5, '"triggers":' '[' '{' '"github":' '{' '"secret":' '"e8Z9PWAmgWTl5ahU8Ek6"' '},' '"type":' '"GitHub"' '},' '{' '"generic":' '{' '"secret":' '"qq7bCqRTNJKY9Kt9IKPt"' '},' '"type":' '"Generic"' '},' '{' '"type":' '"ConfigChange"' '},' '{' '"imageChange":' '{' '"lastTriggeredImageID":' '"centos/ruby-22-centos7@sha256:a18c8706118a5c4c9f1adf045024d2abf06ba632b5674b23421019ee4d3edcae"' '},' '"type":' '"ImageChange"' '}' ']' '},' '"status":' '{' '"lastVersion":' 1 '}' '}' '],' '"kind":' '"List",' '"metadata":' '{' '"resourceVersion":' '"",' '"selfLink":' '""' '}' '}'
+ jq 'del(.items[].status,.items[].metadata.uid,.items[].metadata.selfLink,.items[].metadata.resourceVersion,.items[].metadata.generation,.items[].triggers[].imageChangeParams.lastTriggeredImage)'
jq: error (at <stdin>:1): Cannot iterate over null (null)

gerald1248 commented 6 years ago

@e-minguez I've fixed the bug you discovered. One of the jq expressions was missing the leading ".spec".

e-minguez commented 6 years ago

I've tested and it seems to be working fine. Thanks!

openshift / openshift-ansible-contrib