search

DevelopingSpace / starchart

A self-serve tool for managing custom domains and certificates
MIT License
20 stars 13 forks source link

Reconciler crashes with Missing field 'SetIdentifier' #446

Closed humphd closed 1 year ago

humphd commented 1 year ago

I tried running the reconciler on staging, and it crashes with this in the server logs (cc @dadolhay):

starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    | {"changeSet":[{"Action":"DELETE","ResourceRecordSet":{"MultiValueAnswer":false,"Name":"stage.mystudentproject.ca.","ResourceRecords":[{"Value":"\"v=spf1 -all\""}],"TTL":300,"Type":"TXT"}},{"Action":"DELETE","ResourceRecordSet":{"MultiValueAnswer":false,"Name":"_dmarc.stage.mystudentproject.ca.","ResourceRecords":[{"Value":"\"v=DMARC1; p=reject; sp=reject\""}],"TTL":300,"Type":"TXT"}},{"Action":"DELETE","ResourceRecordSet":{"MultiValueAnswer":false,"Name":"foo.davidhumphrey.stage.mystudentproject.ca.","ResourceRecords":[{"Value":"github.com"}],"TTL":300,"Type":"CNAME"}},{"Action":"DELETE","ResourceRecordSet":{"MultiValueAnswer":false,"Name":"test1.wno.stage.mystudentproject.ca.","ResourceRecords":[{"Value":"192.168.0.1"}],"TTL":300,"Type":"A"}}],"level":"debug","message":"Reconciler intends to push the following 4 changes","timestamp":"2023-03-25T23:43:50.576Z"}
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    | {"error":{"$fault":"client","$metadata":{"attempts":1,"httpStatusCode":400,"requestId":"07d2e314-a5a1-4c94-9a2d-9a12c9ff0350","totalRetryDelay":0},"Code":"InvalidInput","Type":"Sender","name":"InvalidInput"},"level":"error","message":"DNS Error - Failed to execute changeSet","timestamp":"2023-03-25T23:43:50.742Z"}
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    | InvalidInput: Invalid request: Missing field 'SetIdentifier' in Change with [Action=DELETE, Name=stage.mystudentproject.ca., Type=TXT, SetIdentifier=null]
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     at deserializeAws_restXmlInvalidInputResponse (/app/node_modules/@aws-sdk/client-route-53/dist-cjs/protocols/Aws_restXml.js:5262:23)
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     at deserializeAws_restXmlChangeResourceRecordSetsCommandError (/app/node_modules/@aws-sdk/client-route-53/dist-cjs/protocols/Aws_restXml.js:1928:25)
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     at processTicksAndRejections (node:internal/process/task_queues:95:5)
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     at /app/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     at /app/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:14:20
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     at /app/node_modules/@aws-sdk/middleware-retry/dist-cjs/retryMiddleware.js:27:46
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     at /app/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     at executeChangeSet (/app/build/index.js:769:20)
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     at reconcile (/app/build/index.js:2696:7)
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     at action7 (/app/build/index.js:2712:14) {
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |   '$fault': 'client',
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |   '$metadata': {
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     httpStatusCode: 400,
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     requestId: '07d2e314-a5a1-4c94-9a2d-9a12c9ff0350',
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     extendedRequestId: undefined,
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     cfId: undefined,
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     attempts: 1,
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     totalRetryDelay: 0
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |   },
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |   Type: 'Sender',
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |   Code: 'InvalidInput'
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    | }

Here is is cleaned up:

{
  "changeSet": [
    {
      "Action": "DELETE",
      "ResourceRecordSet": {
        "MultiValueAnswer": false,
        "Name": "stage.mystudentproject.ca.",
        "ResourceRecords": [{ "Value": "\"v=spf1 -all\"" }],
        "TTL": 300,
        "Type": "TXT"
      }
    },
    {
      "Action": "DELETE",
      "ResourceRecordSet": {
        "MultiValueAnswer": false,
        "Name": "_dmarc.stage.mystudentproject.ca.",
        "ResourceRecords": [{ "Value": "\"v=DMARC1; p=reject; sp=reject\"" }],
        "TTL": 300,
        "Type": "TXT"
      }
    },
    {
      "Action": "DELETE",
      "ResourceRecordSet": {
        "MultiValueAnswer": false,
        "Name": "foo.davidhumphrey.stage.mystudentproject.ca.",
        "ResourceRecords": [{ "Value": "github.com" }],
        "TTL": 300,
        "Type": "CNAME"
      }
    },
    {
      "Action": "DELETE",
      "ResourceRecordSet": {
        "MultiValueAnswer": false,
        "Name": "test1.wno.stage.mystudentproject.ca.",
        "ResourceRecords": [{ "Value": "192.168.0.1" }],
        "TTL": 300,
        "Type": "A"
      }
    }
  ],
  "level": "debug",
  "message": "Reconciler intends to push the following 4 changes",
  "timestamp": "2023-03-25T23:43:50.576Z"
}

{
  "error": {
    "$fault": "client",
    "$metadata": {
      "attempts": 1,
      "httpStatusCode": 400,
      "requestId": "07d2e314-a5a1-4c94-9a2d-9a12c9ff0350",
      "totalRetryDelay": 0
    },
    "Code": "InvalidInput",
    "Type": "Sender",
    "name": "InvalidInput"
  },
  "level": "error",
  "message": "DNS Error - Failed to execute changeSet",
  "timestamp": "2023-03-25T23:43:50.742Z"
}

And:

InvalidInput: Invalid request: Missing field 'SetIdentifier' in Change with [Action=DELETE, Name=stage.mystudentproject.ca., Type=TXT, SetIdentifier=null]

I'm not finding a specific answer to why this is happening in my searches so far.

It looks like there are records in here that aren't related to our app, so we're going to have to rethink the idea of owning the entire zone.

humphd commented 1 year ago

My guess is that SetIdentifier is set on one of those records we don't own, and it's refusing to delete them because we haven't included it where it expects to find it.

I wonder if we should limit our scope to only records that follow the form {something}.{someusername}.root.domain., which would eliminate both stage.mystudentproject.ca. and _dmarc.stage.mystudentproject.ca. in the case above.

humphd commented 1 year ago

Despite the work to add limp mode in #447, I'd like to avoid stepping on records we don't actually own in the hosted zone.

humphd commented 1 year ago

This was fixed in our debugging tonight.