pulumi-fastly provider failure

[shaidar]

In an effort to standardize the OCW Fastly configs, we had started working on using the pulumi-fastly provider whereby we manually create a Fastly template distribution containing all the necessary OCW configs, importing that into Pulumi and then using the Pulumi code generated to configure the already existing Fastly OCW distributions.

The template distro config:

import pulumi_fastly as fastly

ol_devops_qa = fastly.ServiceVcl("ol-devops-qa",
    activate=True,
    backends=[
        fastly.ServiceVclBackendArgs(
            address="ocw-content-live-qa.s3.us-east-1.amazonaws.com",
            between_bytes_timeout=10000,
            connect_timeout=1000,
            first_byte_timeout=15000,
            max_conn=200,
            name="WebsiteBucket",
            override_host="ocw-content-live-qa.s3.us-east-1.amazonaws.com",
            port=443,
            request_condition="not course media or old akamai",
            ssl_cert_hostname="ocw-content-live-qa.s3.us-east-1.amazonaws.com",
            ssl_check_cert=True,
            ssl_sni_hostname="ocw-content-live-qa.s3.us-east-1.amazonaws.com",
            use_ssl=True,
            weight=100,
        ),
        fastly.ServiceVclBackendArgs(
            address="ocw-website-storage.s3.us-east-1.amazonaws.com",
            between_bytes_timeout=10000,
            connect_timeout=1000,
            first_byte_timeout=15000,
            max_conn=200,
            name="OCWWebsiteStorageBucket",
            override_host="ocw-website-storage.s3.us-east-1.amazonaws.com",
            port=443,
            request_condition="is old Akamai file",
            ssl_cert_hostname="ocw-website-storage.s3.us-east-1.amazonaws.com",
            ssl_check_cert=True,
            ssl_sni_hostname="ocw-website-storage.s3.us-east-1.amazonaws.com",
            use_ssl=True,
            weight=100,
        ),
    ],
    comment="",
    conditions=[
        fastly.ServiceVclConditionArgs(
            name="not course media or old akamai",
            priority=10,
            statement="req.url.path !~ \"^/coursemedia\" && req.url.path !~ \"^/ans\\d+\"",
            type="REQUEST",
        ),
        fastly.ServiceVclConditionArgs(
            name="Generated by synthetic response for 503 page",
            statement="beresp.status == 503",
            type="CACHE",
        ),
        fastly.ServiceVclConditionArgs(
            name="is old Akamai file",
            priority=10,
            statement="req.url.path ~ \"^/ans\\d+\" && req.url.path !~ \"/ans7870/21f/21f.027\"",
            type="REQUEST",
        ),
    ],
    default_ttl=3600,
    dictionaries=[fastly.ServiceVclDictionaryArgs(
        name="redirects",
    )],
    domains=[fastly.ServiceVclDomainArgs(
        name="ol-devops-qa.odl.mit.edu",
    )],
    gzips=[fastly.ServiceVclGzipArgs(
        content_types=[
            "text/html",
            "application/x-javascript",
            "text/css",
            "application/javascript",
            "text/javascript",
            "application/json",
            "application/vnd.ms-fontobject",
            "application/x-font-opentype",
            "application/x-font-truetype",
            "application/x-font-ttf",
            "application/xml",
            None,
            "font/opentype",
            "font/otf",
            "image/svg+xml",
            "image/vnd.microsoft.icon",
            "text/plain",
            "text/xml",
        ],
        extensions=[
            None,
            None,
            "js",
            None,
            "html",
            "ico",
            None,
            "otf",
            "svg",
        ],
        name="Generated by default gzip policy",
    )],
    logging_s3s=[
        fastly.ServiceVclLoggingS3Args(
            bucket_name="ol-ocw-site-logs",
            name="OCW Fastly Logs",
            path="fastly/qa/%Y%M/",
            s3_iam_role="arn:aws:iam::610119931565:role/ol-ocw-site-fastly-role",
            timestamp_format="%Y-%m-%dT%H:%M:%S",
            format='{ "timestamp": "%{strftime(\{"%Y-%m-%dT%H:%M:%S%z"\}, time.start)}V", "client_ip": "%{req.http.Fastly-Client-IP}V", "geo_country": "%{client.geo.country_name}V", "geo_city": "%{client.geo.city}V", "host": "%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V", "url": "%{json.escape(req.url)}V", "request_method": "%{json.escape(req.method)}V", "request_protocol": "%{json.escape(req.proto)}V", "request_referer": "%{json.escape(req.http.referer)}V", "request_user_agent": "%{json.escape(req.http.User-Agent)}V", "response_state": "%{json.escape(fastly_info.state)}V", "response_status": %{resp.status}V, "response_reason": %{if(resp.response, "%22"+json.escape(resp.response)+"%22", "null")}V, "response_body_size": %{resp.body_bytes_written}V, "fastly_server": "%{json.escape(server.identity)}V", "fastly_is_edge": %{if(fastly.ff.visits_this_service == 0, "true", "false")}V }',
        ),
    ],
    headers=[
        fastly.ServiceVclHeaderArgs(
            action="set",
            source="beresp.http.x-amz-meta-site-id",
            destination="http.Surrogate-Key",
            name="S3 Cache Surrogate Keys",
            priority=10,
            type="cache",
        ),
        fastly.ServiceVclHeaderArgs(
            action="set",
            source="max-age=300",
            destination="http.Strict-Transport-Security",
            name="Generated by force TLS and enable HSTS",
            priority=100,
            type="response",
        ),
    ],
    name="ol-devops-qa",
    request_settings=[fastly.ServiceVclRequestSettingArgs(
        force_ssl=True,
        name="Generated by force TLS and enable HSTS",
    )],
    response_objects=[fastly.ServiceVclResponseObjectArgs(
        cache_condition="Generated by synthetic response for 503 page",
        content="""<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8">
    <title>503</title>
  </head>
  <body>
    503
  </body>
</html>""",
        content_type="text/html",
        name="Generated by synthetic response for 503 page",
        response="Service Unavailable",
        status=503,
    )],
    snippets=[
        fastly.ServiceVclSnippetArgs(
            content="""if (obj.status == 601 && obj.response == "redirect") {
  set obj.status = 302;
  set obj.http.Location = req.protocol + "://" + req.http.host + req.http.header_sans_department;
  return (deliver);
}

if (obj.status == 602 && obj.response == "redirect") {
  set obj.status = 302;
  set obj.http.Location = req.protocol + "://" + req.http.host + req.http.pages_header;
  return (deliver);
}

if (obj.status == 301) {
  set obj.status = 302;
  set obj.http.Location = req.protocol + "://" + req.http.host + req.http.slash_header;
  return (deliver);
}""",
            name="Reroute Redirects",
            priority=100,
            type="error",
        ),
        fastly.ServiceVclSnippetArgs(
            content="""table departments BOOL {
  "chemistry": true,
  "biological-engineering": true,
  "aeronautics-and-astronautics": true,
  "global-studies-and-languages": true,
  "mathematics": true,
  "health-sciences-and-technology": true,
  "edgerton-center": true,
  "womens-and-gender-studies": true,
  "urban-studies-and-planning": true,
  "sloan-school-of-management": true,
  "anthropology": true,
  "engineering-systems-division": true,
  "chemical-engineering": true,
  "earth-atmospheric-and-planetary-sciences": true,
  "music-and-theater-arts": true,
  "athletics-physical-education-and-recreation": true,
  "architecture": true,
  "experimental-study-group": true,
  "literature": true,
  "mechanical-engineering": true,
  "electrical-engineering-and-computer-science": true,
  "materials-science-and-engineering": true,
  "history": true,
  "linguistics-and-philosophy": true,
  "institute-for-data-systems-and-society": true,
  "biology": true,
  "civil-and-environmental-engineering": true,
  "comparative-media-studies-writing": true,
  "nuclear-engineering": true,
  "economics": true,
  "concourse": true,
  "brain-and-cognitive-sciences": true,
  "science-technology-and-society": true,
  "political-science": true,
  "media-arts-and-sciences": true,
  "physics": true,
  "supplemental-resources": true
}""",
            name="Departments Table",
            priority=100,
            type="init",
        ),
        fastly.ServiceVclSnippetArgs(
            content="""# Remove AWS headers returned from S3
unset resp.http.x-amz-id-2;
unset resp.http.x-amz-request-id;
unset resp.http.x-amz-version-id;
unset resp.http.x-amz-meta-s3cmd-attrs;
unset resp.http.server;

# Remove unnecessary headers that add weight
unset resp.http.via;
unset resp.http.x-timer;""",
            name="Clean response headers on delivery",
            priority=100,
            type="deliver",
        ),
        fastly.ServiceVclSnippetArgs(
            content="""declare local var.location STRING;
declare local var.status INTEGER;
declare local var.last_path_part STRING;
declare local var.redirect STRING;
declare local var.url STRING;

# Redirect bare directory names to add a trailing slash
if (req.url.path ~ "/([^/]+)$") {               // ends with non-slash character
  set var.last_path_part = re.group.1;
  if (var.last_path_part !~ "\.[a-zA-Z0-9]{1,4}$") {  // is directory, not file w. extension
    set req.http.slash_header = req.url.path + "/";
    if (std.strlen(req.url.qs) > 0) {
      set req.http.slash_header = req.http.slash_header + "?" + req.url.qs;
    }
    error 301 req.http.slash_header;
  }
}

# Perform redirects with dictionary lookups
set var.url = regsub(req.url.path, "/$", "");
set var.redirect = table.lookup(redirects, var.url);

if (var.redirect ~ "^([0-9]{3})\|([^|]*)\|(.*)$") {

  set var.status = std.atoi(re.group.1);

  if (std.strlen(req.url.qs) > 0) {
    set var.location = re.group.3 + if(re.group.2 == "keep", "?" + req.url.qs, "");
  } else {
    set var.location = re.group.3;
  }
  set var.location = regsub(var.location, "{{AK_HOSTHEADER}}", req.http.host);
  set var.location = regsub(var.location, "{{PMUSER_PATH}}", "");

  error var.status var.location;
}

# OCW Legacy Department Redirects
if (req.url.path ~ "(/courses/)([\w-]+)/(.*)/") {
  if (table.lookup_bool(departments, re.group.2, false)) {
    if (std.strlen(re.group.4) > 0) {
      set req.http.header_sans_department = re.group.1 + re.group.3 + "/" + re.group.4;
    } else {
      set req.http.header_sans_department = re.group.1 + re.group.3 + "/";
    }
    error 601 "redirect";
  }
}""",
            name="Redirects",
            type="recv",
        ),
        fastly.ServiceVclSnippetArgs(
            content=""" if (beresp.status == 404) {
  set beresp.ttl = 5m;
  set beresp.http.Cache-Control = "max-age=300";
  return(deliver);
}

if (bereq.url.path ~ "^/main\.[0-9a-f]+\.(css|js)") {
    set beresp.ttl = 2629743s;  // one month
    set beresp.http.Cache-Control = "max-age=2629743";
}""",
            name="TTLs setup",
            priority=100,
            type="fetch",
        ),
        fastly.ServiceVclSnippetArgs(
            content=""" # Add /pages/ to URL in case of 404
if (beresp.status == 404) {
  if (req.url.path ~ "(/courses/)([\w-]+)/([\w-]+)/(.*)") {
    if (std.strlen(re.group.4) > 0) {
      set req.http.pages_header = re.group.1 + re.group.2 + "/resources/" + re.group.4;
    } else {
      set req.http.pages_header = re.group.1 + re.group.2 + "/pages/" + re.group.3;
    }
    error 602 "redirect";
  }
}""",
            name="Legacy OCW Pages Redirect",
            priority=100,
            type="fetch",
        ),
        fastly.ServiceVclSnippetArgs(
            content="""if (req.url.ext == "css") {
  set beresp.http.Content-type = "text/css";
}

if (req.url.ext == "js") {
  set beresp.http.Content-type = "application/javascript";
}

if (req.url.ext == "png") {
  set beresp.http.Content-type = "image/png";
}

if (req.url.ext == "jpg") {
  set beresp.http.Content-type = "image/jpeg";
}

if (req.url.ext == "gif") {
  set beresp.http.Content-type = "image/gif";
}

if (req.url.ext == "pdf") {
  set beresp.http.Content-type = "application/pdf";
}""",
            name="Set correct Content-type for S3 assets",
            priority=100,
            type="fetch",
        ),
        fastly.ServiceVclSnippetArgs(
            content="""set bereq.url = querystring.remove(bereq.url);
set bereq.url = regsub(bereq.url, "/$", "/index.html");
set bereq.url = regsub(bereq.url, "^/ans7870", "/largefiles");
set bereq.url = regsub(bereq.url, "^/ans15436", "/zipfiles");
#set bereq.url = querystring.remove(bereq.url);""",
            name="S3 Bucket Proxying",
            priority=100,
            type="miss",
        ),
    ],
    stale_if_error=True,
    stale_if_error_ttl=43200,
    opts=pulumi.ResourceOptions(protect=True))

Things I tried:

• Using pulumi-fastly 4.0.0, tried importing an existing distro (ol-devops-ci), added logging or made a minor change and then running pulumi up, and it failed with the following error: missing required field 'ServiceVersion' This appears to be a known issue with the latest version.

• Tried downgrading to different pulumi-fastly provider versions (3.7.0, 3.11.0, 3.12.0) which uses the fastly.Servicev1 as opposed to fastly.ServiceVcl being now used in version 4.0.0 however the not so descriptive error I was encountering was: error: unknown resource type '<nil>'

• Using pulumi-fastly 4.0.0, I was able to create a new distro (based on the config above) and that was successful. However, when I tried tweaking anything in that config and running pulumi up again, I encountered the ServiceVersion error and when I ran it again, it was successful, however when checking on the Fastly config, it was still running the old version and the change wasn't deployed.

• Additionally when doing all this testing, there were a few times when pulumi up succeeded however when I checked on the Fastly distro, it was still on the old version and no changes had been deployed. In that sense it was quite misleading and not reliable to be used in a prod environment.

Solution

• I commented on the existing Github issue, posted in pulumi slack and Fastly slack channels to hopefully have someone look into it as we currently are blocked on using this until we feel confident about the results.

[shaidar]

Tested out new version of pulumi-fastly provider (4.0.1) and ran into the same issue. Ran pulumi up with the minimal example below. It created the Fastly distro successfully, however after tweaking the code by changing the domain comment, it threw the infamous ServiceVersion error

# Original minimal example

import pulumi
import pulumi_fastly as fastly

ol_devops_qa = fastly.ServiceVcl("ol-devops-qa",
    backends=[fastly.ServiceVclBackendArgs(
        address="127.0.0.1",
        name="localhost",
        port=80,
    )],
    domains=[fastly.ServiceVclDomainArgs(
        comment="ol-devops-qa",
        name="ol-devops-qa.odl.mit.edu",
    )],
    force_destroy=True)

# Tweaked minimal example by changing comment

import pulumi
import pulumi_fastly as fastly

ol_devops_qa = fastly.ServiceVcl("ol-devops-qa",
    backends=[fastly.ServiceVclBackendArgs(
        address="127.0.0.1",
        name="localhost",
        port=80,
    )],
    domains=[fastly.ServiceVclDomainArgs(
        comment="ol-devops-test",
        name="ol-devops-qa.odl.mit.edu",
    )],
    force_destroy=True)

[blarghmatey]

This is still blocked until https://github.com/pulumi/pulumi-fastly/issues/119 is resolved

[shaidar]

Issue has been resolved and am currently importing the existing OCW Fastly distro configs into Pulumi. Had to import both ServiceVcl and ServiceDictionaryItems to cover all the configured resources. Now pulumi state has the following fastly configs imported:

• ocw-draft.QA
• ocw-live.QA
• ocw-draft.Production
• ocw-live.Production

Will not work on templatising the output so that we have the Fastly configs as Pulumi code and going forward we can ensure that changes go through the normal code change process to avoid config drift.