search

ducktors / turborepo-remote-cache

Open source implementation of the Turborepo custom remote cache server.
https://ducktors.github.io/turborepo-remote-cache/
MIT License
1.03k stars 95 forks source link

Artifacts are never uploaded to cache server with AWS Lambda deployment #148

Closed zdarovka closed 1 year ago

zdarovka commented 1 year ago

What version of Turborepo are you using?

1.8.3

What package manager are you using / does the bug impact?

Yarn v1, Yarn v2/v3 (node_modules linker only)

What operating system are you using?

Mac

Describe the Bug

The PUT endpoint for uploading artifacts is never being called when deployed to AWS lambda, when configured for Vercel servers it just stores them fine.

My config.json

{
  "teamId": "team_FcALQN9XEVbeJ1NjTQoS9Wup",
  "apiUrl": "https://XXXXXXX.execute-api.eu-central-1.amazonaws.com",
  "token": "abcd"
}

Note: I have tried both teamId and teamSlug for the team identifier. It does not make any change.

My build command

yarn turbo run build --token="abcd"

(yes, I am passing token twice to really make sure it gets there)

Requests on the cache server

1 GET Artefact

Request

{
    "severity": "INFO",
    "level": 30,
    "time": 1677700599229,
    "pid": 8,
    "hostname": "169.254.44.209",
    "reqId": "LuNKSAAKR5e9t5CU_gvE2A-2",
    "req": {
        "method": "GET",
        "url": "/v8/artifacts/3f04bb2017b72da6?teamId=team_FcALQN9XEVbeJ1NjTQoS9Wup",
        "hostname": "XXXXX.execute-api.eu-central-1.amazonaws.com",
        "remoteAddress": "XXXXX"
    },
    "message": "incoming request"
}

Response

{
    "severity": "WARNING",
    "level": 40,
    "time": 1677700599257,
    "pid": 8,
    "hostname": "XXX",
    "reqId": "LuNKSAAKR5e9t5CU_gvE2A-2",
    "data": {
        "message": null,
        "code": "BadRequest",
        "region": null,
        "time": "2023-03-01T19:56:39.239Z",
        "requestId": "G1XS8G2FKHWSXTSZ",
        "extendedRequestId": "XCmY4Ggr4L/bmuBfzffH445xoSDEb5qa/Dxo/lvYUanFAZtHoj9lY0Wq2xK/fH2VylFyByXiHsE=",
        "statusCode": 400,
        "retryable": false,
        "retryDelay": 40.85058254148659
    },
    "isBoom": true,
    "isServer": false,
    "output": {
        "statusCode": 404,
        "payload": {
            "statusCode": 404,
            "error": "Not Found",
            "message": "Artifact not found"
        },
        "headers": {}
    },
    "stack": "Error: Artifact not found\n    at Object.handler (/var/task/index.js:100897:37)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)",
    "type": "Error",
    "message": "Artifact not found"
}

POST artefact events

Request

{
    "severity": "INFO",
    "level": 30,
    "time": 1677700599517,
    "pid": 8,
    "hostname": "XXXX",
    "reqId": "LuNKSAAKR5e9t5CU_gvE2A-3",
    "req": {
        "method": "POST",
        "url": "/v8/artifacts/events?teamId=team_FcALQN9XEVbeJ1NjTQoS9Wup",
        "hostname": "XXXX.execute-api.eu-central-1.amazonaws.com",
        "remoteAddress": "86.49.226.196"
    },
    "message": "incoming request"
}

Response

{
    "severity": "INFO",
    "level": 30,
    "time": 1677700599518,
    "pid": 8,
    "hostname": "XXXX",
    "reqId": "LuNKSAAKR5e9t5CU_gvE2A-3",
    "res": {
        "statusCode": 200
    },
    "responseTime": 1.0016169999726117,
    "message": "request completed"
}

No other reuqest follows this sequence, and the result is artefacts are not uploaded

Your Environment

Expected Behavior

Artefacts should be uploaded.

fox1t commented 1 year ago

Hi! Can you please run yarn turbo run build --token="abcd" --remote-only --dry-run json This should add more information about the error.

P.S. The token field inside the config.json is not used by the turbo client.

zdarovka commented 1 year ago

Here it is 

 yarn turbo run build --token="abcd" --remote-only --dry-run json --filter public
{
  "globalHashSummary": {
    "globalFileHashMap": {},
    "rootExternalDepsHash": "f3cbb08864837d4d",
    "globalCacheKey": "Buffalo buffalo Buffalo buffalo buffalo buffalo Buffalo buffalo",
    "pipeline": {
      "build": {
        "outputs": [
          ".next/**",
          "_static/**",
          "dist/**"
        ],
        "cache": true,
        "dependsOn": [
          "^build"
        ],
        "inputs": [],
        "outputMode": "full",
        "env": [
          "CF_BUCKET",
          "CF_URL",
          "NEXT_PUBLIC_API_HOST"
        ],
        "persistent": false
      },
      "dev": {
        "outputs": [],
        "cache": false,
        "dependsOn": [],
        "inputs": [],
        "outputMode": "full",
        "env": [],
        "persistent": true
      },
      "export": {
        "outputs": [
          "_static/**"
        ],
        "cache": true,
        "dependsOn": [
          "build"
        ],
        "inputs": [],
        "outputMode": "full",
        "env": [],
        "persistent": false
      },
      "generate:api": {
        "outputs": [],
        "cache": false,
        "dependsOn": [
          "^generate:api"
        ],
        "inputs": [],
        "outputMode": "full",
        "env": [],
        "persistent": false
      },
      "lint": {
        "outputs": [],
        "cache": true,
        "dependsOn": [],
        "inputs": [],
        "outputMode": "full",
        "env": [],
        "persistent": false
      },
      "start": {
        "outputs": [],
        "cache": true,
        "dependsOn": [],
        "inputs": [],
        "outputMode": "full",
        "env": [],
        "persistent": true
      }
    }
  },
  "packages": [
    "public"
  ],
  "tasks": [
    {
      "taskId": "tsconfig#build",
      "task": "build",
      "package": "tsconfig",
      "hash": "635373fad73ef9d7",
      "cacheState": {
        "local": false,
        "remote": false
      },
      "command": "\u003cNONEXISTENT\u003e",
      "outputs": [
        ".next/**",
        "_static/**",
        "dist/**"
      ],
      "excludedOutputs": null,
      "logFile": "packages/tsconfig/.turbo/turbo-build.log",
      "directory": "packages/tsconfig",
      "dependencies": [],
      "dependents": [
        "api#build",
        "public#build",
        "translations#build",
        "ui#build"
      ],
      "resolvedTaskDefinition": {
        "outputs": [
          ".next/**",
          "_static/**",
          "dist/**"
        ],
        "cache": true,
        "dependsOn": [
          "^build"
        ],
        "inputs": [],
        "outputMode": "full",
        "env": [
          "CF_BUCKET",
          "CF_URL",
          "NEXT_PUBLIC_API_HOST"
        ],
        "persistent": false
      }
    },
    {
      "taskId": "eslint-config-custom#build",
      "task": "build",
      "package": "eslint-config-custom",
      "hash": "e42f0fc95d479d6c",
      "cacheState": {
        "local": false,
        "remote": false
      },
      "command": "\u003cNONEXISTENT\u003e",
      "outputs": [
        ".next/**",
        "_static/**",
        "dist/**"
      ],
      "excludedOutputs": null,
      "logFile": "packages/eslint-config-custom/.turbo/turbo-build.log",
      "directory": "packages/eslint-config-custom",
      "dependencies": [],
      "dependents": [
        "api#build",
        "public#build",
        "translations#build",
        "ui#build"
      ],
      "resolvedTaskDefinition": {
        "outputs": [
          ".next/**",
          "_static/**",
          "dist/**"
        ],
        "cache": true,
        "dependsOn": [
          "^build"
        ],
        "inputs": [],
        "outputMode": "full",
        "env": [
          "CF_BUCKET",
          "CF_URL",
          "NEXT_PUBLIC_API_HOST"
        ],
        "persistent": false
      }
    },
    {
      "taskId": "translations#build",
      "task": "build",
      "package": "translations",
      "hash": "b286163c43a50da0",
      "cacheState": {
        "local": false,
        "remote": false
      },
      "command": "\u003cNONEXISTENT\u003e",
      "outputs": [
        ".next/**",
        "_static/**",
        "dist/**"
      ],
      "excludedOutputs": null,
      "logFile": "packages/translations/.turbo/turbo-build.log",
      "directory": "packages/translations",
      "dependencies": [
        "tsconfig#build",
        "eslint-config-custom#build"
      ],
      "dependents": [
        "public#build"
      ],
      "resolvedTaskDefinition": {
        "outputs": [
          ".next/**",
          "_static/**",
          "dist/**"
        ],
        "cache": true,
        "dependsOn": [
          "^build"
        ],
        "inputs": [],
        "outputMode": "full",
        "env": [
          "CF_BUCKET",
          "CF_URL",
          "NEXT_PUBLIC_API_HOST"
        ],
        "persistent": false
      }
    },
    {
      "taskId": "api#build",
      "task": "build",
      "package": "api",
      "hash": "7b264fb7dbc70b7b",
      "cacheState": {
        "local": false,
        "remote": false
      },
      "command": "\u003cNONEXISTENT\u003e",
      "outputs": [
        ".next/**",
        "_static/**",
        "dist/**"
      ],
      "excludedOutputs": null,
      "logFile": "packages/api/.turbo/turbo-build.log",
      "directory": "packages/api",
      "dependencies": [
        "tsconfig#build",
        "eslint-config-custom#build"
      ],
      "dependents": [
        "public#build"
      ],
      "resolvedTaskDefinition": {
        "outputs": [
          ".next/**",
          "_static/**",
          "dist/**"
        ],
        "cache": true,
        "dependsOn": [
          "^build"
        ],
        "inputs": [],
        "outputMode": "full",
        "env": [
          "CF_BUCKET",
          "CF_URL",
          "NEXT_PUBLIC_API_HOST"
        ],
        "persistent": false
      }
    },
    {
      "taskId": "ui#build",
      "task": "build",
      "package": "ui",
      "hash": "d028f979b1e8c173",
      "cacheState": {
        "local": false,
        "remote": false
      },
      "command": "\u003cNONEXISTENT\u003e",
      "outputs": [
        ".next/**",
        "_static/**",
        "dist/**"
      ],
      "excludedOutputs": null,
      "logFile": "packages/ui/.turbo/turbo-build.log",
      "directory": "packages/ui",
      "dependencies": [
        "eslint-config-custom#build",
        "tsconfig#build"
      ],
      "dependents": [
        "public#build"
      ],
      "resolvedTaskDefinition": {
        "outputs": [
          ".next/**",
          "_static/**",
          "dist/**"
        ],
        "cache": true,
        "dependsOn": [
          "^build"
        ],
        "inputs": [],
        "outputMode": "full",
        "env": [
          "CF_BUCKET",
          "CF_URL",
          "NEXT_PUBLIC_API_HOST"
        ],
        "persistent": false
      }
    },
    {
      "taskId": "public#build",
      "task": "build",
      "package": "public",
      "hash": "795cfa2ada484105",
      "cacheState": {
        "local": false,
        "remote": false
      },
      "command": "next build",
      "outputs": [
        ".next/**",
        "_static/**",
        "dist/**"
      ],
      "excludedOutputs": null,
      "logFile": "apps/public/.turbo/turbo-build.log",
      "directory": "apps/public",
      "dependencies": [
        "ui#build",
        "translations#build",
        "api#build",
        "tsconfig#build",
        "eslint-config-custom#build"
      ],
      "dependents": [],
      "resolvedTaskDefinition": {
        "outputs": [
          ".next/**",
          "_static/**",
          "dist/**"
        ],
        "cache": true,
        "dependsOn": [
          "^build"
        ],
        "inputs": [],
        "outputMode": "full",
        "env": [
          "CF_BUCKET",
          "CF_URL",
          "NEXT_PUBLIC_API_HOST"
        ],
        "persistent": false
      }
    }
  ]
}
cvle commented 1 year ago

We have the same or similar issue. We see only the tiniest artefacts (around 15kB) being uploaded, but not larger ones. I can only imagine that there is some background process running that is not properly awaited, because AWS Lambda would terminate and not wait for any background processes to finish when they are not awaited. That's only a theory though, without diving deeper into the implementation.

zdarovka commented 1 year ago

We have the same or similar issue. We see only the tiniest artefacts (around 15kB) being uploaded, but not larger ones. I can only imagine that there is some background process running that is not properly awaited, because AWS Lambda would terminate and not wait for any background processes to finish when they are not awaited. That's only a theory though, without diving deeper into the implementation.

I don't think that is my case. I do not see any incoming request to my lambda which would suggest that local turbo is trying to upload anything. Also I do see a message "Remote cashing enabled" but at the end of the built, there is actually just a super quick message about writing to cache, the local one, but nothing more heavier going out of my machine.

zdarovka commented 1 year ago

I have created a new turborepo repo from scratch where I am experiencing the same problem. No cache artefacts are being uploaded, and no error is shown. Feel fry to try it with my cache server.

yarn turbo run build --token="abcd" --remote-only

https://github.com/zdarovka/demo-turbo-repo

Dry run 

{
  "globalHashSummary": {
    "globalFileHashMap": {},
    "rootExternalDepsHash": "d9edb4fa4a52bc4c",
    "globalCacheKey": "Buffalo buffalo Buffalo buffalo buffalo buffalo Buffalo buffalo",
    "pipeline": {
      "build": {
        "outputs": [
          ".next/**",
          "dist/**"
        ],
        "cache": true,
        "dependsOn": [
          "^build"
        ],
        "inputs": [],
        "outputMode": "full",
        "env": [],
        "persistent": false
      },
      "dev": {
        "outputs": [],
        "cache": false,
        "dependsOn": [],
        "inputs": [],
        "outputMode": "full",
        "env": [],
        "persistent": false
      },
      "lint": {
        "outputs": [],
        "cache": true,
        "dependsOn": [],
        "inputs": [],
        "outputMode": "full",
        "env": [],
        "persistent": false
      }
    }
  },
  "packages": [
    "docs",
    "eslint-config-custom",
    "tsconfig",
    "ui",
    "web"
  ],
  "tasks": [
    {
      "taskId": "eslint-config-custom#build",
      "task": "build",
      "package": "eslint-config-custom",
      "hash": "4e579143b2440ab5",
      "cacheState": {
        "local": false,
        "remote": false
      },
      "command": "\u003cNONEXISTENT\u003e",
      "outputs": [
        ".next/**",
        "dist/**"
      ],
      "excludedOutputs": null,
      "logFile": "packages/eslint-config-custom/.turbo/turbo-build.log",
      "directory": "packages/eslint-config-custom",
      "dependencies": [],
      "dependents": [
        "docs#build",
        "ui#build",
        "web#build"
      ],
      "resolvedTaskDefinition": {
        "outputs": [
          ".next/**",
          "dist/**"
        ],
        "cache": true,
        "dependsOn": [
          "^build"
        ],
        "inputs": [],
        "outputMode": "full",
        "env": [],
        "persistent": false
      }
    },
    {
      "taskId": "tsconfig#build",
      "task": "build",
      "package": "tsconfig",
      "hash": "01b087a0d8a547f8",
      "cacheState": {
        "local": false,
        "remote": false
      },
      "command": "\u003cNONEXISTENT\u003e",
      "outputs": [
        ".next/**",
        "dist/**"
      ],
      "excludedOutputs": null,
      "logFile": "packages/tsconfig/.turbo/turbo-build.log",
      "directory": "packages/tsconfig",
      "dependencies": [],
      "dependents": [
        "docs#build",
        "ui#build",
        "web#build"
      ],
      "resolvedTaskDefinition": {
        "outputs": [
          ".next/**",
          "dist/**"
        ],
        "cache": true,
        "dependsOn": [
          "^build"
        ],
        "inputs": [],
        "outputMode": "full",
        "env": [],
        "persistent": false
      }
    },
    {
      "taskId": "ui#build",
      "task": "build",
      "package": "ui",
      "hash": "9260002178eae286",
      "cacheState": {
        "local": false,
        "remote": false
      },
      "command": "\u003cNONEXISTENT\u003e",
      "outputs": [
        ".next/**",
        "dist/**"
      ],
      "excludedOutputs": null,
      "logFile": "packages/ui/.turbo/turbo-build.log",
      "directory": "packages/ui",
      "dependencies": [
        "tsconfig#build",
        "eslint-config-custom#build"
      ],
      "dependents": [
        "docs#build",
        "web#build"
      ],
      "resolvedTaskDefinition": {
        "outputs": [
          ".next/**",
          "dist/**"
        ],
        "cache": true,
        "dependsOn": [
          "^build"
        ],
        "inputs": [],
        "outputMode": "full",
        "env": [],
        "persistent": false
      }
    },
    {
      "taskId": "web#build",
      "task": "build",
      "package": "web",
      "hash": "dfad8c743259d84b",
      "cacheState": {
        "local": false,
        "remote": false
      },
      "command": "next build",
      "outputs": [
        ".next/**",
        "dist/**"
      ],
      "excludedOutputs": null,
      "logFile": "apps/web/.turbo/turbo-build.log",
      "directory": "apps/web",
      "dependencies": [
        "ui#build",
        "tsconfig#build",
        "eslint-config-custom#build"
      ],
      "dependents": [],
      "resolvedTaskDefinition": {
        "outputs": [
          ".next/**",
          "dist/**"
        ],
        "cache": true,
        "dependsOn": [
          "^build"
        ],
        "inputs": [],
        "outputMode": "full",
        "env": [],
        "persistent": false
      }
    },
    {
      "taskId": "docs#build",
      "task": "build",
      "package": "docs",
      "hash": "6110f25d79dfafa1",
      "cacheState": {
        "local": false,
        "remote": false
      },
      "command": "next build",
      "outputs": [
        ".next/**",
        "dist/**"
      ],
      "excludedOutputs": null,
      "logFile": "apps/docs/.turbo/turbo-build.log",
      "directory": "apps/docs",
      "dependencies": [
        "ui#build",
        "eslint-config-custom#build",
        "tsconfig#build"
      ],
      "dependents": [],
      "resolvedTaskDefinition": {
        "outputs": [
          ".next/**",
          "dist/**"
        ],
        "cache": true,
        "dependsOn": [
          "^build"
        ],
        "inputs": [],
        "outputMode": "full",
        "env": [],
        "persistent": false
      }
    }
  ]
}

Build log

petrzdarsky@Petrs-MacBook-Pro my-turborepo % yarn turbo run build --token="abcd" --remote-only -vvv
yarn run v1.22.19
$ /Users/petrzdarsky/Downloads/demo-turbo-repo/my-turborepo/node_modules/.bin/turbo run build --token=abcd --remote-only -vvv
2023-03-02T21:26:29.436+0100 [DEBUG] turborepo_lib::shim: Global turbo version: 1.8.3
2023-03-02T21:26:29.440+0100 [DEBUG] turborepo_lib::shim: Local turbo version: 1.8.3
2023-03-02T21:26:29.440+0100 [DEBUG] turborepo_lib::shim: Repository Root: /Users/petrzdarsky/Downloads/demo-turbo-repo/my-turborepo
2023-03-02T21:26:29.440+0100 [DEBUG] turborepo_lib::shim: Running local turbo binary in /Users/petrzdarsky/Downloads/demo-turbo-repo/my-turborepo/node_modules/turbo/bin/turbo

2023-03-02T21:26:29.440+0100 [DEBUG] turborepo_lib::shim: supports_skip_infer_and_single_package true
• Packages in scope: docs, eslint-config-custom, tsconfig, ui, web
• Running build in 5 packages
• Remote caching enabled
web:build: Skipping cache check for web#build, outputs have not changed since previous run.
web:build: cache hit, replaying output dfad8c743259d84b
web:build: $ next build
web:build: info  - Linting and checking validity of types...
web:build: info  - Creating an optimized production build...
web:build: info  - Compiled successfully
web:build: info  - Collecting page data...
web:build: info  - Generating static pages (0/3)
web:build: info  - Generating static pages (3/3)
web:build: info  - Finalizing page optimization...
web:build: 
web:build: Route (pages)                              Size     First Load JS
web:build: ┌ ○ /                                      300 B          73.4 kB
web:build: └ ○ /404                                   181 B          73.3 kB
web:build: + First Load JS shared by all              73.1 kB
web:build:   ├ chunks/framework-243845dae5bb80aa.js   45.2 kB
web:build:   ├ chunks/main-88dbbcb312ed251a.js        27 kB
web:build:   ├ chunks/pages/_app-70fbc1cf4ceebc50.js  195 B
web:build:   └ chunks/webpack-4e7214a60fad8e88.js     712 B
web:build: 
web:build: ○  (Static)  automatically rendered as static HTML (uses no initial props)
web:build: 
docs:build: cache miss, executing 64c204511f457322
docs:build: $ next build
docs:build: info  - Linting and checking validity of types...
docs:build: info  - Creating an optimized production build...
docs:build: info  - Compiled successfully
docs:build: info  - Collecting page data...
docs:build: info  - Generating static pages (0/3)
docs:build: info  - Generating static pages (3/3)
docs:build: info  - Finalizing page optimization...
docs:build: 
docs:build: Route (pages)                              Size     First Load JS
docs:build: ┌ ○ /                                      301 B          73.4 kB
docs:build: └ ○ /404                                   181 B          73.3 kB
docs:build: + First Load JS shared by all              73.1 kB
docs:build:   ├ chunks/framework-243845dae5bb80aa.js   45.2 kB
docs:build:   ├ chunks/main-88dbbcb312ed251a.js        27 kB
docs:build:   ├ chunks/pages/_app-70fbc1cf4ceebc50.js  195 B
docs:build:   └ chunks/webpack-4e7214a60fad8e88.js     712 B
docs:build: 
docs:build: ○  (Static)  automatically rendered as static HTML (uses no initial props)
docs:build: 

 Tasks:    2 successful, 2 total
Cached:    1 cached, 2 total
  Time:    10.284s 

✨  Done in 10.78s.
zdarovka commented 1 year ago

I have also just tried the free server in Europe, and I am still unable to save the cache.

Server Status TURBO_API TURBO_TOKEN
Europe   https://eu.trrc.ducktors.dev turbotoken-oss-europe1
zdarovka commented 1 year ago

I think it is all related due to lambda request body size limits

https://github.com/cvle/turborepo-cache-lambda/issues/1

matteovivona commented 1 year ago

Thanks for looking into it further. you might consider opening an issue on that repository