Migration should notify of skipped resources

[stevendborrelli]

What problem are you facing?

When creating a migration on a local filesystem if a file contains unknown Kinds, the migrator will not process the file without emitting any warnings or errors.

For example, if a manifest contains a Secret which does not have a registered converter, outputs are never generated.

---
apiVersion: v1
kind: Secret
metadata:
  name: demo-foo-mq-rabbitmq-creds
  namespace: example
  annotations:
    argocd.argoproj.io/sync-wave: "-2"
type: Opaque
immutable: true
data:
  password: "cGFzc3dvcmQK"
---
apiVersion: mq.aws.crossplane.io/v1alpha1
kind: Broker
metadata:
  labels:
    crossplane.io/provider-version: v0.26.1
  name: demo-foo-mq-rabbitmq
  namespace: example
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  forProvider:
    region: us-east-1
    engineType: RabbitMQ
    publiclyAccessible: false
    autoMinorVersionUpgrade: false
    engineVersion: 3.8.11
    hostInstanceType: mq.t3.micro
    deploymentMode: SINGLE_INSTANCE
    subnetIDs:   
      - subnet-2cc5ce61
    securityGroupIdRefs:
      - name: demo-foo-mq-rabbitmq-security-group
    users:
      - username: rabbitmq-admin
        consoleAccess: true
        passwordSecretRef:
          key: password
          name: demo-foo-mq-rabbitmq-creds
          namespace: example
    tags:
      App: crossplane
      Environment: qa
      Name: crossplane-mq
      Product: crossplane
      Squad: sandbox
      Tier: backend
      VS: sandbox
      cluster: mq-cluster
  writeConnectionSecretToRef:
    name: demo-foo-mq-creds
    namespace: example
  providerConfigRef:
    name: awsconfig

The generated migration plan is mostly empty and none of the other directories are crated.

spec: {}
version: 0.1.0

How could Upjet help solve your problem?

The user should be informed of any local filesystem resources that have been skipped during processing.

[stevendborrelli]

Another way to create this is have a resource with an error in a file with multiple MRs to convert.

For example, if you have 2 Queues and the 2nd has an error, only the first will be processed. Only when removing the first resource will the error appear:

---
apiVersion: sqs.aws.crossplane.io/v1beta1
kind: Queue
metadata:
  labels:
    crossplane.io/provider-version: v0.26.1
  name: sqs-simple-helm-2-dlq
  annotations:
    argocd.argoproj.io/sync-wave: "4"
spec:
  forProvider:
    region: us-east-1
    delaySeconds: 4
    maximumMessageSize: 262144
    messageRetentionPeriod: 345600
    receiveMessageWaitTimeSeconds: 0
    visibilityTimeout: 30
    sseEnabled: true

    policy: |
      {
        "Version": "2012-10-17",
        "Id": "sqs-simple-helm-2",
        "Statement": [
          {
            "Sid": "BasicSQS",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
              "sqs:DeleteMessage",
              "sqs:SendMessage"
            ],
            "Resource": "arn:aws:sqs:us-east-1::sqs-simple-helm-2",
            "Condition": {
              "StringLikeIfExists": {
                "aws:SourceAccount": ""
              }
            }
          }
        ]
      }

    tags:
      Name: sqs-simple-helm-2-dlq
      VS: sandbox
  providerConfigRef:
    name: awsconfig
---
apiVersion: sqs.aws.crossplane.io/v1beta1
kind: Queue
metadata:
  labels:
    crossplane.io/provider-version: v0.26.1
  name: sqs-simple-helm-2
  namespace:
  annotations:
    argocd.argoproj.io/sync-wave: "5"
spec:
  forProvider:
    region: us-east-1
    delaySeconds: 4
    maximumMessageSize: 262144
    messageRetentionPeriod: 345600
    receiveMessageWaitTimeSeconds: 0
    visibilityTimeout: 30

    sseEnabled: true

    policy: |
      {
        "Version": "2012-10-17",
        "Id": "sqs-simple-helm-2",
        "Statement": [
          {
            "Sid": "BasicSQS",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
              "sqs:DeleteMessage",
              "sqs:SendMessage"
            ],
            "Resource": "arn:aws:sqs:us-east-1::sqs-simple-helm-2",
            "Condition": {
              "StringLikeIfExists": {
                "aws:SourceAccount": ""
              }
            }
          }
        ]
      }
    redrivePolicy:
      deadLetterTargetArnRef:
        name: sqs-simple-helm-2-dlq
      maxReceiveCount: 1

    tags:
      Name: sqs-simple-helm-2
      VS: sandbox
  providerConfigRef:
    name: awsconfig

go run cmd/migrator/main.go -s ./source Using filesystem source main: error: Failed to generate the migration plan: failed to generate the migration plan: failed to migrate resource: failed to migrate resource: failed to copy source into target: failed to convert from unstructured.Unstructured to the managed resource type: unrecognized type: string

[haooliveira84]

I think this issue is also related with https://github.com/upbound/upjet/issues/233