Open florianmaxim opened 4 years ago
I'm afraid there're some cases that I didn't find in the Twilio official docs. For example:
This bodySHA256
handling:
https://github.com/twilio/twilio-node/blob/565cc4cc523ab7fe7efa3a9eb2dcc73bfa55e759/lib/webhooks/webhooks.js#L170-L184 https://github.com/twilio/twilio-node/blob/565cc4cc523ab7fe7efa3a9eb2dcc73bfa55e759/lib/webhooks/webhooks.js#L62-L64
This port
handling:
Is there any port
or bodySHA256
stuff on your received request?
Hm, do you maybe see any issue in my nginx handler ?
`location /api/ {
proxy_pass_header Server;
proxy_set_header Host $http_host;
proxy_set_header X-Scheme $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_redirect off;
proxy_connect_timeout 20;
proxy_read_timeout 20;
proxy_pass https://nodejs:3000/;
}
`
@chentsulin Ah, wait it always comes with a second error:
ErrorUrl | "https://xxxxxx.dev/api/webhooks/whatsapp" |
---|---|
SmsMessageSid | "SM849483eda06exxxxxx" |
ErrorCode | "11200" |
NumMedia | "0" |
SmsSid | "SM849483eda06e945xxxxxx" |
SmsStatus | "received" |
Body | "hallo" |
To | "whatsapp:+14155xxxxxx" |
NumSegments | "1" |
MessageSid | "SM849483eda06e945d69exxxxxx" |
AccountSid | "ACd5227f8efd4687bb2xxxxxx" |
From | "whatsapp:+491764xxxxxx" |
ApiVersion | "2010-04-01" |
sourceComponent | "14100" |
---|---|
url | "null" |
ErrorCode | "11200" |
LogLevel | "ERROR" |
Msg | "The URI scheme, of the URI null, must be equal (ignoring case) to 'http', 'https', 'ws', or 'wss'" |
EmailNotification | "false" |
For the first error, if you're using express behind nginx you may want to enable('trust proxy')
, so your server can use the X-Forwarded-*
headers to determine the connection and the IP address of the client:
https://stackoverflow.com/questions/39930070/nodejs-express-why-should-i-use-app-enabletrust-proxy
For the second error, are you sending any urls (for example, media url) in your app? could you show me the url or the function call?
Oh, maybe I should make some changes to the inferred url: https://github.com/Yoctol/bottender/blob/ce0f13c78dd0d4cdf462b2e9316ef7e1a9278bbe/packages/bottender/src/server/Server.ts#L136
And still need another two nginx configs to make it works:
https://www.nginx.com/resources/wiki/start/topics/examples/forwarded/
X-Forwarded-Host: example.com
X-Forwarded-Proto: https
The server app is as reduced as this:
` const server = express();
server.set('trust proxy');
const verify = (req, _, buf) => { req.rawBody = buf.toString(); }; server.use(bodyParser.json({ verify })); server.use(bodyParser.urlencoded({ extended: false, verify }));
server.all('*', (req, res) => { return handle(req, res); });`
Thank you so much for the update!
I updated bottender to 1.4.7 as well as the NGINX server block:
` proxy_pass http://nodejs:3000/;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
`
But still getting the same error from f*** Twilio :( This is frustrating since everthing is working so well with th eother platforms!
Any other hints?..
Are you able to print this out on your express server? This might helps a lot.
const server = express();
server.use((req, res, next) => {
console.log('headers', JSON.stringify(req.headers));
console.log('hostname', req.hostname);
console.log('protocol', req.protocol);
console.log('requestUrl', `${req.protocol}://${req.hostname}${req.url}`);
next();
});
We should make sure the requestUrl here in your server match the url you registered on Twilio.
here is the log:
`"headers": { "x-real-ip":"54.242.136.XXX", "x-forwarded-for":"54.242.136.XXX", "x-forwarded-proto":"https", "host":"hn-bot.dev", "x-forwarded-host":"XXX.dev", "x-forwarded-port":"443", "connection":"close", "content-length":"318", "content-type":"application/x-www-form-urlencoded", "x-twilio-signature":"9gsseAIhctS18mbe7dErVXXXXXXXXX", "i-twilio-idempotency-token":"8f13d9ea-4884-4b81-8dd9-7XXXXXXXXX", "accept":"/", "cache-control":"max-age=259200", "user-agent":"TwilioProxy/1.1" }"
hostname: XXX.dev protocol: http requestUrl http://XXX.dev/webhooks/whatsapp`
So I guess "x-forwarded-proto":"https"
and protocol: http
should match, right?
It's a little bit weird.
I expected express req.protocol
should return the value of x-forwarded-proto
when trust proxy mode is on.
I see your comment calling server.set('trust proxy');
here:
https://github.com/Yoctol/bottender/issues/815#issuecomment-647446220
If you did that, you should change that line to one of the following call:
server.set('trust proxy', true);
// or
server.enable('trust proxy');
Those two do the exactly same thing. See: https://github.com/expressjs/express/blob/18da651c5b57431f61391d9de2518a8c2ef88b2c/lib/application.js#L452-L454
But you can't call
server.set('trust proxy');
@chentsulin Ah, damn it, my bad! I changed it and now it matches:
"x-forwarded-proto":"https" and protocol: https
matches.
But I just realized the url ending /api/
is getting lost on it's way!
requestUrl http://XXX.dev/webhooks/whatsapp
should actually be requestUrl http://XXX.dev/api/webhooks/whatsapp
Any idea where this is happening?
Could req.originalUrl
help in this case?
const server = express();
server.use((req, res, next) => {
console.log('originalUrl', req.originalUrl);
console.log('url', req.url);
console.log('requestUrl', `${req.protocol}://${req.hostname}${req.originalUrl}`);
next();
});
If /api
is included in req.originalUrl
, we should switch to use it.
I already tried. originalUrl /webhooks/whatsapp
doesnt include the path as well..
It is not supposed to be included in the bottender.config.js
, or is it?
whatsapp: { enabled: true, path: '/webhooks/whatsapp',
How about adding $request_uri
in your nginx config proxy_pass
part?
http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass
Note: you might need to modify path in your bottender.config.js
accordingly
How about adding
$request_uri
in your nginx configproxy_pass
part? http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass
It's already inside a location block:
location /api/ { proxy_pass http://nodejs:3000;
So I'm relatively sure I don't need the $request_uri
in here, or do I?
@tw0517tw
Ah, it has to be included on the proxy header as well, ofc!
proxy_set_header Host $host$request_uri;
proxy_set_header X-Forwarded-Host $host$request_uri;
EDIT: Which is actually not a soluction, since modifiying the host like that doesnt realy make sense...
Okay, maybe this helps:
Since it's all working FINE for other messengers here's the slack config:
The Slack request URL is set to https://xxx.dev/api/webhooks/slack
.
Path in bottender.config.js is set to /webhooks/messenger'.
Enabled trust proxy server.enable('trust proxy');
.
Log of a slack request:
hostname xxx.dev
protocol https
originalUrl /webhooks/slack
requestUrl https://xxx.dev
NGINX block:
location /api/ {
proxy_pass http://nodejs:3000/;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
}
Why is this not working for twilio?
@florianmaxim
Twilio is the only provider that uses HTTP URL to sign the signature. I'm thinking the graceful way to solve this url behind proxy issue, but if you want to work around this, you may try this for now:
const server = express();
server.use((req, res, next) => {
req.url = `/api${req.url}`; // conditionally apply this on your proxy env
next();
});
Edit: Sorry my bad. Should be /api${req.url}
instead of /api/${req.url}
Not everyone have permission to change the settings of nginx, so maybe we should support some other options to config host
, protocol
and url
just like twilio-node
does:
@florianmaxim
Twilio is the only provider that uses HTTP URL to sign the signature. I'm thinking the graceful way to resolve this url behind proxy issue, but if you want to work around this, you may try this for now:
const server = express(); server.use((req, res, next) => { req.url = `/api${req.url}`; // conditionally apply this on your proxy env next(); });
Edit: Sorry my bad. Should be
/api${req.url}
instead of/api/${req.url}
That indeed turn the log console.log('requestUrl',
${req.protocol}://${req.hostname}${req.url});
into https://xxx.dev/api/webhooks/whatsapp
.
But results into a 502 on Twilio:
Twilio was unable to fetch content from: http://xxx.dev/api/webhooks/whatsapp
Error: Total timeout is triggered. Configured tt is 15000ms and we attempted 1 time(s)
Account SID: ACd5227f8efd4687bb211da6e5xxxxxx
SID: SM5f8958810d5a3c070db2f889fxxxxxx
Request ID: 5864b582-3ca8-41bd-923e-5ed38xxxxxx
Remote Host: xxx.dev
Request Method: POST
Request URI: http://xxx.dev/api/webhooks/whatsapp
URL Fragment: true
:/ :(
After adding that path patch, you need to modify path in your bottender.config.js
accordingly. For example:
whatsapp: { enabled: true, path: '/webhooks/whatsapp',
to
whatsapp: { enabled: true, path: '/api/webhooks/whatsapp',
Note: This affects every channel (messenger, whatsapp...).
After adding that path patch, you need to modify path in your
bottender.config.js
accordingly. For example:whatsapp: { enabled: true, path: '/webhooks/whatsapp',
to
whatsapp: { enabled: true, path: '/api/webhooks/whatsapp',
Note: This affects every channel (messenger, whatsapp...).
Right, of course! This works.
So briefly summarized:
server.use((req, res, next) => {
if(req.originalUrl === '/webhooks/whatsapp') {
req.url = `/<url_path>${req.url}`;
}
}):
whatsapp: {
enabled: true,
path: '/<url_path>/webhooks/whatsapp',
}
Thanks a lot to @tw0517tw @chentsulin for this quick workaround!!
Hey there! Thank you for this great project! Unfortunalety I'm having trouble connecting twilio (live credentials) to my whatsapp webhook (other webhooks work great).. Twilio is trowing an "11200" errorCode containing this body:
{ "error": { "message": "WhatsApp Signature Validation Failed!", "request": { "rawBody": "xxx", "headers": { "x-twilio-signature": "xxxLzNwapQY4rGkb+mVFYO2I2jI=" } } } }
Any help would would be appreciated!