localstack / airspeed

A Python implementation of Velocity templates, tailored to VTL in AWS (e.g., API Gateway, AppSync)
Other
2 stars 1 forks source link

Add initial setup for snapshot testing against AWS and Java VTL #9

Closed whummer closed 1 year ago

whummer commented 1 year ago

Add initial setup for snapshot testing against AWS and Java VTL.

The PR adds some test fixtures to run VTL tests against two additional targets for validation:

It turns out that there are a number of discrepancies between the logic in AWS and the airspeed implementation. For now, the failing tests are annotated with @pytest.mark.xfail - over time we should fix the logic and add switches to ensure our logic is compatible with AWS.

Note: The PR removes support for Python <=3.9 in the build matrix - some module files in localstack core (runtime) are using 3.10 features, and we're using airspeed only in the runtime components (within the LS container), hence no need to support older versions.

/cc @simonrw @bentsku @calvernaz

whummer commented 1 year ago

but the VTL cli message is when airspeed doesn't match the Java VTL, but works according to AWS ?

I suppose snapshot with 2 different targets is kinda difficult... should the only target in the end be AWS?

Good points - the ultimate target (test oracle) in the end should be AWS. Let's keep that in mind for future iterations. 👍 The Java VTL tests should become more of an additional data point to see where and in which cases AWS diverges from the standard.

Btw, this is also one of the reasons why it makes sense to continue with our fork, as opposed to collaborating on the upstream repo, as presumably the original airspeed aims to be as close as possible to the Java VTL implementation, not the AWS flavor. (I've also updated that in the repo description, will add a note to the README as well)

As a next step, I'd go ahead and try to fix some of the parity issues.. (separate PR)

calvernaz commented 1 year ago

but the VTL cli message is when airspeed doesn't match the Java VTL, but works according to AWS ?

I suppose snapshot with 2 different targets is kinda difficult... should the only target in the end be AWS?

Good points - the ultimate target (test oracle) in the end should be AWS. Let's keep that in mind for future iterations. +1 The Java VTL tests should become more of an additional data point to see where and in which cases AWS diverges from the standard.

Yep, I like the data point for sure, I was purely asking for understanding if the existing marker highlights a mistmatch with the CLI but it's working from AWS perspective