search

dherault / serverless-offline

Emulate AWS λ and API Gateway locally when developing your Serverless project
MIT License
5.2k stars 794 forks source link

"serverless-offline" v6 alpha feedback #768

Closed dnalborczyk closed 4 years ago

dnalborczyk commented 5 years ago

Thank you for giving serverless-offline v6 alpha a try! 🎉

In this issue thread you can provide feedback regarding ongoing work for v6.

The current v6 releases are mainly marked as an alpha version to signal that stuff might be breaking, due to major code refactoring, reasons listed below (breaking changes), as well as the current lack of tests (which hopefully will improve over time). If you encounter any bugs please open a new issue stating the alpha version you are using.

Installation

npm i serverless-offline@next --save-dev

Latest release

v6.0.0 alpha.66

New features

Planned new features

Maybe new features

Improvements

Breaking changes (see below for a migration path)

Temporary Breaking changes

Bugfixes

Performance

Other

Migration Paths

--binPath option has been removed

we're not invoking serverless invoke local anymore, instead we moved the (lightly modified) code into the plugin and calling it ourselves. this is also way faster: ~ 25 times (on a mac, with python)

--exec option has been removed

just call your script as part of a shell script, npm script, javascript etc.

why was this option removed?

--location option has been removed

if your handler file has different locations depending on the environment.

instead, use custom variables!

e.g.

custom:
  location:
    development: some-other-path/src
    production: src

function:
  hello:
    handler: ${self:custom.location.${opt:stage}}.hello
    # or alternatively, whichever you prefer
    # handler: ${self:custom.location.${env:NODE_ENV}}.hello

why was this option removed?

--noEnvironment option has been removed

why was this option removed?

--showDuration option has been removed

nothing needs to be done, it just works

why was this option removed?

--providedRuntime option has been removed

instead, use custom variables!

at provider level, e.g.:

custom:
  runtime:
    development: nodejs10.x # use supported runtime here
    production: provided

provider:
  runtime: ${self:custom.runtime.${opt:stage}}
  # or alternatively, whichever you prefer
  # runtime: ${self:custom.runtime.${env:NODE_ENV}}

or at function level, e.g.:

custom:
  runtime:
    development: nodejs10.x # use supported runtime here
    production: provided

function:
  hello:
    handler: handler.hello
    runtime: ${self:custom.runtime.${opt:stage}} # or alternatively ${env:NODE_ENV}
    # or alternatively, whichever you prefer
    # runtime: ${self:custom.runtime.${env:NODE_ENV}}

why was this option removed?

--region option has been removed

coming soon

--stage option has been removed

coming soon

--prefix option has been removed

coming soon

--preserverTrailingSlash option has been removed

coming soon

event.isOffline=true has been removed

process.env.IS_OFFLINE=true has been removed

or in the function definition

functions: hello: handler: handler.hello environment: IS_OFFLINE: true



**why was this option removed?**
- for parity with the AWS event object

#### remove functionality shared with https://github.com/svdgraaf/serverless-plugin-stage-variables 

_coming soon_

_This list will be updated with each alpha release._
computerpunc commented 5 years ago

@dnalborczyk What would be the recommended way of pushing updates to v6 alpha?

I’m thinking of:

  1. Fixing #355 and #678.
  2. Moving all current WebSocket tests Into integration phase so everyone (including CI) can run WebSocket tests and not just me :)
dnalborczyk commented 5 years ago

@computerpunc awesome!

@dnalborczyk What would be the recommended way of pushing updates to v6 alpha?

just do a PR against master. I didn't branch off, as it originally wasn't planned to be what it became. there's a 5.x-branch for v5 releases, but I'd rather do that only if absolutely necessary, as the code base diverged too much.

I’m thinking of:

  1. Fixing #355 and #678.

sure, go for it! it would be great if you could also add some tests, similar to: https://github.com/dherault/serverless-offline/tree/master/src/__tests__/integration/handler

I also want to move everything non-unit test out of the "__tests__" folders, as those should be only used for unit tests.

I'm also planning an additional (optional) test script to automatically deploy to AWS and run the same tests against the deployment and expect the same results. that would be most likely done only locally by any developers or users, unless we have or get an AWS test account.

regarding #678 I introduced a LambdaFunction class, which also runs the lambda handler. unit tests are included. (unit tests for exceptions I have in a stash, not in master yet). we probably eventually need some generic version of how to reply to 'function errors' with 'lambda', 'proxy', 'websocket' etc.

  1. Moving all current WebSocket tests Into integration phase so everyone (including CI) can run WebSocket tests and not just me :)

that would be great!

btw, I wanna look at the websocket-authorizer merge later today so we can merge into master soon. maybe I'll move the test folders afterwards, otherwise it causes more friction.

computerpunc commented 5 years ago

@dnalborczyk

Python integration test fails.

python tests › should work with python

    FetchError: invalid json response body at http://localhost:3000/hello reason: Unexpected end of JSON input
dnalborczyk commented 5 years ago

hey @computerpunc

thank you! just fixed it. seems like it was only failing on windows because shebangs are not supported, and it tried to run serverless with the windows scripting host and not node.

computerpunc commented 5 years ago

@dnalborczyk

I'm running on OSX. Now both Ruby and Python fail. This is latest plain vanilla master.

python tests › should work with python

    FetchError: invalid json response body at http://localhost:3000/hello reason: Unexpected end of JSON input

ruby tests › should work with ruby

    FetchError: invalid json response body at http://localhost:3000/hello reason: Unexpected end of JSON input
dnalborczyk commented 5 years ago

@computerpunc

you probably have to run npm ci locally. I added a new dependency: execa, to fix spawning a process on windows. let me know if that fixes it.

computerpunc commented 5 years ago

@dnalborczyk

you probably have to run npm ci locally.

It didn't fully help. Python still fails. Below is the errors generated:

Serverless: GET /hello (λ: hello)
Proxy Handler could not detect JSON:

Proxy Handler could not detect JSON:   Error --------------------------------------------------

  spawn python2 ENOENT

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information ---------------------------
     Operating System:          darwin
     Node Version:              12.7.0
     Serverless Version:        1.49.0
     Enterprise Plugin Version: 1.3.8
     Platform SDK Version:      2.1.0
dnalborczyk commented 5 years ago

I'm running on OSX.

me too. plus the tests on travis are passing: https://travis-ci.org/dherault/serverless-offline/builds/570274311

I believe ruby and python are pre-installed on macOS.

I'm not a python nor a ruby user, but we should probably also check if python and ruby are installed on the machine where the tests are to be run, and if not, skip them. I'm gonna look into this.

could you run: python --version and ruby --version on your box just to make sure?

also, could you run the following: go to: __tests__/integration/python

and then run: npx serverless invoke local -f hello

computerpunc commented 5 years ago

@dnalborczyk

Ruby test runs OK. Python doesn't but I checked on another Mac it runs OK.

could you run: python --version and ruby --version on your box just to make sure?

On the problematic Mac Python 2.7.10 and on the OK Mac Python 2.7.16.

and then run: npx serverless invoke local -f hello

Same error as before (on the problematic Mac).

computerpunc commented 5 years ago

In addition, I tried to push changes of #772 on a Mac that can run all tests without a problem. Nevertheless, when trying to push via github client, I get the same error.

python-error
dnalborczyk commented 5 years ago

ah, ok. thanks for checking!

just noticed:

spawn python2 ENOENT

I guess it can't find python2 -> python2 --version probably fails.

dnalborczyk commented 5 years ago

@computerpunc should be fixed now

computerpunc commented 5 years ago

@dnalborczyk

I still cannot push any update because the push is rejected (from a Mac that runs the tests without any error). Github Desktop says that both python2 and python3 tests fail. Where exactly do tests run when I try to push updates of a PR? Is it locally or remotely?

dnalborczyk commented 5 years ago

I still cannot push any update because the push is rejected (from a Mac that runs the tests without any error). Github Desktop says that both python2 and python3 tests fail.

it's just a git hook, you can run:

git push --no-verify

to get around it. not always recommended, but in this case a shortcut until we figure out what's wrong. not sure how to do --no-verify with github desktop, and also not sure why github desktop behaves different in that case.

Where exactly do tests run when I try to push updates of a PR? Is it locally or remotely?

local only

dnalborczyk commented 5 years ago

@computerpunc I can also add an ENV flag and/or npm script to skip 'flaky' tests - if that helps?

computerpunc commented 5 years ago

I managed to push. I didn't need to use --no-verify because from the CLI the tests don't fail only with Github Desktop.

remarcable commented 5 years ago

Hello guys, it seems like the Readme has an incorrect npm install: Instead of 

npm i serverless@next --save-dev

it should probably be

npm i serverless-offline@next --save-dev
dnalborczyk commented 5 years ago

Hello guys, it seems like the Readme has an incorrect npm install: Instead of

npm i serverless@next --save-dev

it should probably be

npm i serverless-offline@next --save-dev

thank you @lightningboss ! Fixed.

computerpunc commented 5 years ago

@dnalborczyk

I see that for testing #781, you added another folder (python-big-json) in ./__tests__/integration.

That's going to be very messy if for each serverless.yml there will be a new folder.

serverless has the ability to consume configuration files with different names and locations.

In #732 and #778 (/src/__tests__/manual/websocket), I use the command sls offline --config ./config/serverless.$1.yml in which $1 can be main, authorizer or RouteSelection to run tests from .yml stored in the same folder.

I hope this helps.

dnalborczyk commented 5 years ago

@computerpunc

That's going to be very messy if for each serverless.yml there will be a new folder.

yeah, you are right. that was just a quick drop-in without thinking about the folder structure. that needs a proper rethinking and reorg for sure.

serverless has the ability to consume configuration files with different names and locations.

In #732 and #778 (/src/tests/manual/websocket), I use the command sls offline --config ./config/serverless.$1.yml in which $1 can be main, authorizer or RouteSelection to run tests from .yml stored in the same folder.

I hope this helps.

yeah, thanks for the tip!

I tried to use serverless.yml sub configuration files but couldn't get it to work at the time: https://github.com/dherault/serverless-offline/blob/master/__tests__/integration/handler/serverless.yml#L15

I might have to go back and have another look at it.

verekia commented 5 years ago

EDIT: Created an issue instead of posting the whole thing here: #787 Sorry about that!

dnalborczyk commented 5 years ago

EDIT: Created an issue instead of posting the whole thing here: #787 Sorry about that!

@verekia no worries, you did the right thing. a new issue is better for tracking!

computerpunc commented 5 years ago

@dnalborczyk

While working on #788, I found that Python 3 tests fail on AWS. Run command (after sls deploy from folder ./__tests__/integration/python3):

npm --endpoint=https://xxxxxxxx.execute-api.us-east-1.amazonaws.com/dev run test __tests__/integration/python3/python3.test.js

Output:

> serverless-offline@6.0.0-alpha.15 test .../serverless-offline
> jest --verbose --silent --runInBand "__tests__/integration/python3/python3.test.js"

 FAIL  __tests__/integration/python3/python3.test.js
  Python 3 tests
    ✕ should work with python 3 (549ms)

  ● Python 3 tests › should work with python 3

    expect(received).toEqual(expected) // deep equality

    - Expected
    + Received

      Object {
    -   "message": "Hello Python 3!",
    +   "message": "Forbidden",
      }

      56 |       const response = await fetch(url)
      57 |       const json = await response.json()
    > 58 |       expect(json).toEqual(expected)
         |                    ^
      59 |     })
      60 |   })
      61 | })

      at Object.toEqual (__tests__/integration/python3/python3.test.js:58:20)

Test Suites: 1 failed, 1 total
Tests:       1 failed, 1 total
Snapshots:   0 total
Time:        1.31s, estimated 3s
dnalborczyk commented 5 years ago

While working on #788, I found that Python 3 tests fail on AWS.

thanks for the info @computerpunc ! gonna check it out!

dnalborczyk commented 5 years ago

@computerpunc It seems the npm endpoint stuff wasn't defined in that file. I fixed it. https://github.com/dherault/serverless-offline/commit/a57cc6dfa144227f9121249c4c35726639b6210f

computerpunc commented 5 years ago

@computerpunc It seems the npm endpoint stuff wasn't defined in that file. I fixed it. a57cc6d

@dnalborczyk

Sorry for not being clear enough, the error I provided above is from running #788 where I already added myself 'endpoint' support in the code. Checking it again on master with your update, the outcome is the same error.

dnalborczyk commented 5 years ago

Sorry for not being clear enough, the error I provided above is from running #788 where I already added myself 'endpoint' support in the code. Checking it again on master with your update, the outcome is the same error.

ah, thanks again! the path is wrong https://github.com/dherault/serverless-offline/blob/master/__tests__/integration/python/python3/python3.test.js#L64

I'm fixing some stuff around tests right now. I'm also gonna change the npm_config stuff to use a standard env variable, as it is more commonly used and understood, plus, not sure if yarn would support it.

computerpunc commented 5 years ago

ah, thanks again! the path is wrong https://github.com/dherault/serverless-offline/blob/master/__tests__/integration/python/python3/python3.test.js#L64

Yes, I missed it too.

I'm fixing some stuff around tests right now. I'm also gonna change the npm_config stuff to use a standard env variable, as it is more commonly used and understood, plus, not sure if yarn would support it.

@dnalborczyk can you check #788 and merge it if OK? All the big changes make it very difficult keep PRs up-to-date with merges. If a PR is merged then there's no hassle because it's already in, but if it's waiting it goes out of date as #732 did :(

dnalborczyk commented 5 years ago

can you check #788 and merge it if OK? @computerpunc

it's very high on my list. just needed to fix the python and ruby bugs yesterday, because it felt like getting out of control. now just touching up the test setup (fairly small) so we can implement more tests.

I believe I looked at the HEAD stuff before once, I probably have the stash still waiting around somewhere. I just remember it was a bit simpler, but might have not covered every use case. I try to find it and then we can compare.

btw, running the tests against AWS works great!!! only thing we need is some sort of auto deployment script for all (or individual) serverless.yml which we could use in a setup and teardown process. do you have something like that in place (or similar)?

computerpunc commented 5 years ago

btw, running the tests against AWS works great!!! only thing we need is some sort of auto deployment script for all (or individual) serverless.yml which we could use in a setup and teardown process. do you have something like that in place (or similar)?

@dnalborczyk

I don't have any specific script as I deployed manually but take a look at this folder which has all kind of scripts for similar tasks (on local machine): https://github.com/computerpunc/serverless-offline/tree/websocket-fixes-authorizer/src/__tests__/manual/websocket/scripts

In any case, I hope you are not going to make git test with AWS upon every commit :)

computerpunc commented 5 years ago

@dnalborczyk

Check the integration tests again as they do not work with AWS. In order to fix, you can: + const { pathname } = url + url.pathname =`${pathname}${pathname === '/' ? '' : '/'}${path}` AWS_ENPOINT => AWS_ENDPOINT

computerpunc commented 5 years ago

@dnalborczyk

I've checked and it now works. Thanks.

If you want to greatly improve the running time of tests when running with AWS, move requiring serverless and serverless-offline after the if in setup() in setupTeardown.js as shown below:

  if (RUN_TEST_AGAINST_AWS) {
    return
  }

  const Serverless = require('serverless') // eslint-disable-line global-require
  const ServerlessOffline = require('../../../src/ServerlessOffline.js') // eslint-disable-line global-require
  const serverless = new Serverless({ servicePath })

require('serverless') and require('../../../src/ServerlessOffline.js') take a lot of time to load and there's no reason to pay this penalty if a server already exists (such as the case of AWS).

EDIT: BTW it also makes a very big improvement when creating a serverless-offline server in the tests, i.e when AWS_ENDPOINT is NULL when running npm run test (although I don't understand why).

Jordan-Eckowitz commented 5 years ago

I updated to v6.0.0-alpha.20 and Python endpoints are now failing: https://github.com/dherault/serverless-offline/issues/742#issuecomment-524661827

ozsay commented 5 years ago

i have another issue with webpack (followup of #787).

it seems that serverless-offline doesn't pick up the changes after a bundle is being rebuilt.

I see:

Built at: 08/26/2019 4:43:24 PM
              Asset      Size        Chunks             Chunk Names
    src/handler1.js  4.11 KiB  src/handler1             src/handler1
src/handler1.js.map  3.96 KiB  src/handler1             src/handler1
    src/handler2.js  4.13 KiB  src/handler2  [emitted]  src/handler2
src/handler2.js.map  3.98 KiB  src/handler2  [emitted]  src/handler2
Entrypoint src/handler1 = src/handler1.js src/handler1.js.map
Entrypoint src/handler2 = src/handler2.js src/handler2.js.map
[./src/handler1.js] 293 bytes {src/handler1}
[./src/handler2.js] 306 bytes {src/handler2} [built]
Serverless: Watching for changes...

but the actual code that is running is not the one after the build.

I can reproduce a minimal example if needed.

now I realized that it takes ~1 minute to pick up the changes but it's not consistent...

ozsay commented 5 years ago

another bug in 6 alpha regarding env vars: It looks like process.env accessed outside of the handler is receiving different value (or no value).

This outputs true in 6 but false in 5 with the same configuration (A is declared in the yml to a value):

const A = process.env.A;

const handler = (event, context, callback) => {
  callback(null, {
    statusCode: 200,
    headers: {
      'Access-Control-Allow-Origin': '*',
      'Access-Control-Allow-Credentials': true,
    },
    body: JSON.stringify(A !== process.env.A, null, 2),
  })
}

module.exports = { handler };
dnalborczyk commented 5 years ago

@computerpunc

If you want to greatly improve the running time of tests when running with AWS, move requiring serverless and serverless-offline after the if in setup() in setupTeardown.js as shown below:

  if (RUN_TEST_AGAINST_AWS) {
    return
  }

  const Serverless = require('serverless') // eslint-disable-line global-require
  const ServerlessOffline = require('../../../src/ServerlessOffline.js') // eslint-disable-line global-require
  const serverless = new Serverless({ servicePath })

require('serverless') and require('../../../src/ServerlessOffline.js') take a lot of time to load and there's no reason to pay this penalty if a server already exists (such as the case of AWS).

thank you @computerpunc ! great idea!! it makes sense, since jest is creating a new module reference for each individual test module.

EDIT: BTW it also makes a very big improvement when creating a serverless-offline server in the tests, i.e when AWS_ENDPOINT is NULL when running npm run test (although I don't understand why).

not sure either, other than 1 less require.

in general, there's probably lots of room for improvements regarding the tests. e.g., right now, all tests are running in-band, meaning not in parallel, because of the currently one port we are using. if we'd use random ports for the tests, they would likely run faster as well. the AWS tests could always run in parallel, since they are pointing to different endpoints anyways.

dnalborczyk commented 5 years ago

@Jordan-Eckowitz I answered in the other issue: https://github.com/dherault/serverless-offline/issues/742#issuecomment-524668186

dnalborczyk commented 5 years ago

@ozsay I created 2 new issues, for easier tracking: https://github.com/dherault/serverless-offline/issues/793 and https://github.com/dherault/serverless-offline/issues/794

Jordan-Eckowitz commented 5 years ago

Hi @dnalborczyk ,

I'm still using v6.0.0-alpha.23 and I'm now getting the following error when serverless-offline boots up:

serverless-offline" initialization errored: Cannot find module './misc/polyfills.js
dnalborczyk commented 5 years ago

@Jordan-Eckowitz thank you!

misc folder is missing in the deployed package. I'll fix that!

update: alpha 26 is out, and the issue should be fixed.

Jordan-Eckowitz commented 5 years ago

@Jordan-Eckowitz thank you!

misc folder is missing in the deployed package. I'll fix that!

update: alpha 26 is out, and the issue should be fixed.

Working perfectly now - thanks @dnalborczyk !

computerpunc commented 5 years ago

process.env.IS_OFFLINE=true has been removed

  • you can just provide your own environment variable, in e.g. serverless.yml:
# in the provider definition:
provider:
  environment:
    IS_OFFLINE: true

# or in the function definition
functions:
  hello:
    handler: handler.hello
    environment:
      IS_OFFLINE: true

@dnalborczyk

This piece of documentation is not correct as process.env.IS_OFFLINE was true only when running offline and not on AWS. With the above approach it will be true on both of them.

BTW why did you remove it in the first place?

dnalborczyk commented 5 years ago

@computerpunc

This piece of documentation is not correct as process.env.IS_OFFLINE was true only when running offline and not on AWS. With the above approach it will be true on both of them.

I don't think this is incorrect. Nobody is saying that you should deploy it with this environment set. 😉 Although I admit that it might need further clarification to make more sense, e.g.:

provider:
  environment:
    # pass through env 
    IS_OFFLINE: ${env:IS_OFFLINE}
    # or pass cli option
    IS_OFFLINE: ${opt:IS_OFFLINE}

I'll add this to the migration example. I need to update the migrations steps, they are not up-to-date with the latest breaking changes.

BTW why did you remove it in the first place?

same reason event.isOffline was removed: https://github.com/dherault/serverless-offline/commit/aa7fd511ade5600500dcd2f1047b1d32eaa7a1a6

besides all of the above one could also set process.env.IS_OFFLINE in the cli, but that would not always work depending on how you run your handlers (in-process, worker-thread, child-process). I have it on my to do list to write something up regarding process.env, global state, module state etc.

dherault commented 5 years ago

Congratulation @dnalborczyk on becoming the first contributor of serverless-offline thought so much work. You did an amazing job at implementing new features and refactoring the code. Best.

computerpunc commented 5 years ago

Congratulation @dnalborczyk on becoming the first contributor of serverless-offline thought so much work. You did an amazing job at implementing new features and refactoring the code. Best.

They say that sometime a single picture is better than a 1000 words :)

dnalborczyk

Well done!!!

dnalborczyk commented 5 years ago

@dherault @computerpunc

thank you guys very much!! that's nice to hear!! very much appreciated!! :blush:

justinlazaro-iselect commented 5 years ago

Hi guys, how can i test my defined openapi in serverless.yml?

computerpunc commented 5 years ago

@dnalborczyk

On a the latest master, npm run test reports the following:

Test Suites: 8 failed, 17 passed, 25 total
Tests:       23 failed, 1 skipped, 216 passed, 240 total
Snapshots:   4 passed, 4 total
Time:        77.606s

The 2 main errors are:

ServerlessError: Serverless plugin "./../../../../" initialization errored: Cannot find module '......../serverless-offline/' from 'PluginManager.js'
FetchError: request to http://localhost:3000/xyz failed, reason: connect ECONNREFUSED 127.0.0.1:3000

Do you experience the same?

uldissturms commented 5 years ago

i have another issue with webpack (followup of #787).

it seems that serverless-offline doesn't pick up the changes after a bundle is being rebuilt.

I see:

Built at: 08/26/2019 4:43:24 PM
              Asset      Size        Chunks             Chunk Names
    src/handler1.js  4.11 KiB  src/handler1             src/handler1
src/handler1.js.map  3.96 KiB  src/handler1             src/handler1
    src/handler2.js  4.13 KiB  src/handler2  [emitted]  src/handler2
src/handler2.js.map  3.98 KiB  src/handler2  [emitted]  src/handler2
Entrypoint src/handler1 = src/handler1.js src/handler1.js.map
Entrypoint src/handler2 = src/handler2.js src/handler2.js.map
[./src/handler1.js] 293 bytes {src/handler1}
[./src/handler2.js] 306 bytes {src/handler2} [built]
Serverless: Watching for changes...

but the actual code that is running is not the one after the build.

I can reproduce a minimal example if needed.

now I realized that it takes ~1 minute to pick up the changes but it's not consistent...

  • working with serverless-offline 5.x.x

I'm having the same issue. Is there a known workaround?

leny commented 5 years ago

Hi, having the same issue as ozsay & uldissturms, but I'm using webpack without the serverless-webpack plugin.

I think it's related to the fact that serverless-offline is replicating the cold start mecanism of serverless function.
This is very useful but should be dismissable on demand, like with a flag like --always-cold or something.

The only workaround I fould right now is to exit/restart the serverless offline command, which ios kinda... not fun.

Thanks for your hard work on this useful plugin!