mboudreau
commented
6 years ago @bethesque any ideas? Seems like it's the standalone. Why is it using US-ASCII? Figured everything would be in UTF8 no?
Closed thombergs closed 6 years ago
mboudreau
commented
6 years ago @bethesque any ideas? Seems like it's the standalone. Why is it using US-ASCII? Figured everything would be in UTF8 no?
bethesque
commented
6 years ago What version of everything are you using @thombergs? There was a problem with this that got fixed in a more recent version. Can you please reproduce with the pact-ruby-standalone-e2e-example project?
thombergs
commented
6 years ago @bethesque I will try with the latest versions and if it still fails create a minimal example to reproduce.
mboudreau
commented
6 years ago @thombergs any updates on this?
thombergs
commented
6 years ago @mboudreau @bethesque
Sorry it took so long to get back to you. I finally took the time to create a minimal example. See https://github.com/thombergs/pact-test.
There's a single pact file with a single interaction, that contains a '€' symbol in the body.
When calling npm publish:pacts the file publish-pacts.js is called which publishes the pact file to a Pact Broker.
This task works OK on a windows machine but fails in our unix-based Gitlab CI environment with the error above. Without the € symbol, it works under unix as well.
Our current workaround is replacing some special characters with their unicode escape string (see method replaceSpecialChars in the javascript file). But this is a flakey workaround.
mboudreau
commented
6 years ago amazing, thank you. I'll take a look at this and see what's going on.
bethesque
commented
6 years ago It'll be a problem with the ruby code.
mboudreau
commented
6 years ago That's what I thought, but I'm surprised it happens only on Linux?
On Fri., 16 Mar. 2018, 8:01 am Beth Skurrie, notifications@github.com wrote:
It'll be a problem with the ruby code.
— You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/pact-foundation/pact-node/issues/75#issuecomment-373521479, or mute the thread https://github.com/notifications/unsubscribe-auth/AAjA5OH-Kn43Li59599wlTv5nt7WMSnMks5tetaYgaJpZM4SYB7M .
bethesque
commented
6 years ago Me too. The Travelling Ruby package is different for each of the operating systems however, so it may have to do with the linux one, or some environment variables on the linux one.
bethesque
commented
6 years ago @thombergs could you try setting the environment variable LANG="UTF-8" as per the External encoding section in https://ruby-doc.org/core-2.2.0/Encoding.html ?
thombergs
commented
6 years ago @bethesque doing an export LANG="UTF-8" before running the script doesn't change anything, sadly.
mefellows
commented
6 years ago I'm just doing some refactoring in a pact go and came across a similar regression (on Linux) during provider verification. I'll get it into shape and share here. It once worked, I'm almost positive
mefellows
commented
6 years ago OK, here's the broken build that fails provider verification with the following pact.
{
"consumer": {
"name": "billy"
},
"provider": {
"name": "bobby"
},
"interactions": [
{
"description": "A request to login with user 'billy'",
"providerState": "User billy exists",
"request": {
"method": "POST",
"path": "/users/login",
"headers": {
"Content-Type": "application/json; charset=utf-8"
},
"body": {
"password": "issilly",
"username": "Jean-Marie de La Beaujardière😀😍"
}
},
"response": {
"status": 200,
"headers": {
"Content-Type": "application/json; charset=utf-8"
},
"body": {
"user": {
"name": "Jean-Marie de La Beaujardière😀😍",
"type": "admin"
}
},
"matchingRules": {
"$.body": {
"match": "type"
},
"$.body.user.type": {
"match": "regex",
"regex": "admin|user|guest"
}
}
}
},
{
"description": "A request to login with user 'billy'",
"providerState": "User billy does not exist",
"request": {
"method": "POST",
"path": "/users/login",
"headers": {
"Content-Type": "application/json; charset=utf-8"
},
"body": {
"password": "issilly",
"username": "Jean-Marie de La Beaujardière😀😍"
}
},
"response": {
"status": 404,
"headers": {
"Content-Type": "application/json; charset=utf-8"
}
}
},
{
"description": "A request to login with user 'billy'",
"providerState": "User billy is unauthorized",
"request": {
"method": "POST",
"path": "/users/login",
"headers": {
"Content-Type": "application/json; charset=utf-8"
},
"body": {
"password": "issilly",
"username": "Jean-Marie de La Beaujardière😀😍"
}
},
"response": {
"status": 401,
"headers": {
"Content-Type": "application/json; charset=utf-8"
}
}
}
],
"metadata": {
"pactSpecification": {
"version": "2.0.0"
}
}
}Interestingly, I can't reproduce that issue with my fork at https://github.com/mefellows/pact-ruby-standalone-e2e-example (both Mac OSX and a Docker environment). I wonder if it's because Rack is part of the framework, and therefore the external HTTP request isn't actually being made?
bethesque
commented
6 years ago The standalone e2e executes real HTTP requests. It's this one that doesn't. https://github.com/pact-foundation/pact-ruby-e2e-example
bethesque
commented
6 years ago Isn't it linux that has the issue anyway?
mefellows
commented
6 years ago Hmm good point. Yes it's Linux, but I've not managed to reproduce it even in a Linux docker container with the standalone example from above. So perhaps it's how we're executing the standalone?
The issue is exactly the one from https://github.com/pact-foundation/pact-go/issues/41 and https://github.com/pact-foundation/pact-go/issues/36.
mefellows
commented
6 years ago OK update. Adding Encoding.default_external = 'UTF-8' beneath the requires in pact/lib/vendor/ruby/2.2.0/gems/json-2.1.0/lib/json/common.rb fixes this for my Pact tests. More investigation coming, perhaps it's an env var like LANG configuring it.
mefellows
commented
6 years ago Digger further...
root@3b6b4b83fead:/go/src/github.com/pact-foundation/pact-go# ./build/pact/lib/ruby/bin/ruby -e 'p Encoding.default_external'
#<Encoding:US-ASCII>
root@3b6b4b83fead:/go/src/github.com/pact-foundation/pact-go# LC_ALL=en_US.UTF-8 ./build/pact/lib/ruby/bin/ruby -e 'p Encoding.default_external'
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
#<Encoding:US-ASCII>
root@3b6b4b83fead:/go/src/github.com/pact-foundation/pact-go# LC_ALL= LANG=en_US.UTF-8 ./build/pact/lib/ruby/bin/ruby -e 'p Encoding.default_external'
#<Encoding:US-ASCII>
root@3b6b4b83fead:/go/src/github.com/pact-foundation/pact-go# LC_ALL=C LANG=en_US.UTF-8 ./build/pact/lib/ruby/bin/ruby -e 'p Encoding.default_external'
#<Encoding:US-ASCII>
root@3b6b4b83fead:/go/src/github.com/pact-foundation/pact-go# LC_ALL=en_US.UTF-8 LANG=C ./build/pact/lib/ruby/bin/ruby -e 'p Encoding.default_external'
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
#<Encoding:US-ASCII>Notice how even with the env vars, Ruby still doesn't set encoding!
mefellows
commented
6 years ago OK! Looks like it might be an OS specific thing, which would explain why the standalone test works - it's a slightly different linux variant.
setting LANG=C.UTF-8 fixes the issue for me locally, i'll validate with a Wercker build for Pact Go later.
This now begs the question - should the framework care about this or not? Also, as most wrapper libraries will be kicking off the process execution, we now need to select which, if not all, environment variables come through (my default position will be to pass it through).
mboudreau
commented
6 years ago We should explicitly set it to UTF8 as that should be our baseline.
On Tue, Mar 20, 2018 at 1:03 PM Matt Fellows notifications@github.com wrote:
OK! Looks like it might be an OS specific thing, which would explain why the standalone test works - it's a slightly different linux variant.
setting LANG=C.UTF-8 fixes the issue for me locally, i'll validate with a Wercker build for Pact Go later.
This now begs the question - should the framework care about this or not? Also, as most wrapper libraries will be kicking off the process execution, we now need to select which, if not all, environment variables come through (my default position will be to pass it through).
— You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/pact-foundation/pact-node/issues/75#issuecomment-374447104, or mute the thread https://github.com/notifications/unsubscribe-auth/AAjA5K39FWWYoNUg73hMBi3pUS37Bhohks5tgGNagaJpZM4SYB7M .
bethesque
commented
6 years ago Is it possible that someone may want a different encoding? Wondering if we should set it to UTF-8 if it is not already set?
bethesque
commented
6 years ago @thombergs can you see if setting LANG=C.UTF-8 works for you too?
bethesque
commented
6 years ago My Ubuntu machine has
$ echo $LANG
en_AU.UTF-8Back from lunch - so yeah, it totally fixed it in the latest Pact Go build: https://app.wercker.com/Pact-Foundation/pact-go/runs/build/5ab06ebbdfd4d200017900c7?step=5ab06ee5e368b50001097df7
mboudreau
commented
6 years ago Beth, the web works in UTF8, so I'd doubt anyone would willingly try to get ASCII encoding or windows ones (shudder)
On Tue, Mar 20, 2018 at 2:08 PM Matt Fellows notifications@github.com wrote:
Back from lunch - so yeah, it totally fixed it in the latest Pact Go build: https://app.wercker.com/Pact-Foundation/pact-go/runs/build/5ab06ebbdfd4d200017900c7?step=5ab06ee5e368b50001097df7
— You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/pact-foundation/pact-node/issues/75#issuecomment-374457992, or mute the thread https://github.com/notifications/unsubscribe-auth/AAjA5PCfgWO42BooCLVQR9kz8frC6dhlks5tgHKtgaJpZM4SYB7M .
mefellows
commented
6 years ago That LANG definition looks correct, however in the official golang (Debian stretch) image for example, this is not respected at all. So unfortunately I'm not sure even env vars will solve the problem. I'll investigate a bit later to confirm this, perhaps it's just a matter of ensuring the system is properly configured for a locale.
I'm leaning towards forcing UTF-8, but I'm not yet fully aware of the impact across all tools/languages this might produce.
Documentation might be the next best option.
bethesque
commented
6 years ago I'm inclined to agree with @mboudreau on this one. Let's default it to UTF-8 (I'll set it in the ruby code so that people's default locale won't screw with it), and if someone wants a different encoding, we'll deal with it then via a different mechanism (eg command line parameter). I'm guessing there is a very low chance of anyone wanting a different encoding, as Michel points out.
mefellows
commented
6 years ago I tend to agree. Just as an aside, this issue had encoding set to ASCII-8BIT. So we might need to be agile if it comes up (and probably worth a documentation note across projects).
bethesque
commented
6 years ago I guess the other option is that we could print out a large and ugly warning when the environment variable is anything other than UTF-8. I don't think people tend to read warning messages however!
mefellows
commented
6 years ago FYI I've reproduced this on Windows too (seems to be the activity of the week for us contributors). I just got a working build but removing the UTF-8 stuff. I then tested on a local Windows 10 VM:
PS C:\go\src\github.com\pact-foundation\pact-go> $env:LANG="utf-8"
PS C:\go\src\github.com\pact-foundation\pact-go> C:\pact\pact\lib\ruby\bin\ruby -e 'p Encoding.default_external'
#<Encoding:IBM437>Pretty much everything I tried ended with this.
After manually modifying the file as per above, it worked a treat. I suggest we move to make this happen.
mboudreau
commented
6 years ago @bethesque is the plan to fix it at the ruby standalone by forcing utf8 for everything?
bethesque
commented
6 years ago Yes. I'll add it now.
bethesque
commented
6 years ago Ok, update to the latest standalone.
mboudreau
commented
6 years ago @thombergs Good news, we're now forcing utf8 encoding, get all the tasty goodness from 6.12.2 :)
thombergs
commented
6 years ago @mboudreau @bethesque thanks for the fix! Will try it out in our project as soon as possible.
We currently face an encoding problem while publishing pacts to the pact broker from our npm build on a gitlab CI runner.
The same script works fine from our local windows machines and has worked fine on gitlab (unix), too, until we introduced the "€" symbol into the pacts.
This is the stacktrace:
Any ideas how we can address this issue?