search

apache / openwhisk-package-kafka

Apache OpenWhisk package for communicating with Kafka or Message Hub
https://openwhisk.apache.org/
Apache License 2.0
33 stars 43 forks source link

Remove encoding check #279

Closed abaruni closed 6 years ago

abaruni commented 6 years ago

The data coming from kafka is already utf-8 encoded. it appears as though the encoding check via encode is rejecting valid characters. a similar rejection of valid characters is happening when using decode although happening further up the call stack. this PR is to remove this encoding check altogether.

abaruni commented 6 years ago

Using the kafka-console-producer.sh

Note: 0xb6 is invalid character

>Malalažbeta my weird character \xb6
[2018-08-21T16:47:17.375Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Found 1 messages with a total size of 117 bytes
[2018-08-21T16:47:17.375Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Firing trigger with 1 messages
[2018-08-21T16:47:17.389Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Repsonse status code 202
[2018-08-21T16:47:17.389Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Fired trigger with activation a42be62478264a30abe6247826fa300e
  ➜  openwhisk master ✗   bin/wsk -i --apihost 172.17.0.1 --auth 789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP activation get a42be62478264a30abe6247826fa300e                         
ok: got activation a42be62478264a30abe6247826fa300e
{
    "namespace": "whisk.system",
    "name": "kafkatrigger",
    "version": "0.0.1",
    "subject": "whisk.system",
    "activationId": "a42be62478264a30abe6247826fa300e",
    "start": 1534870037386,
    "end": 0,
    "duration": 0,
    "response": {
        "status": "success",
        "statusCode": 0,
        "success": true,
        "result": {
            "messages": [
                {
                    "key": null,
                    "offset": 44600,
                    "partition": 0,
                    "topic": "test",
                    "value": "Malalažbeta my weird character \\xb6"
                }
            ]
        }
    },
    "logs": [
        "{\"statusCode\":0,\"success\":true,\"activationId\":\"6ca9b6b7b1514c7aa9b6b7b1515c7a86\",\"rule\":\"whisk.system/myrule\",\"action\":\"whisk.system/myecho\"}"
    ],
    "annotations": [],
    "publish": false
}
>"Malalažbeta my weird character \xb6"
[2018-08-21T16:51:00.471Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Found 1 messages with a total size of 121 bytes
[2018-08-21T16:51:00.471Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Firing trigger with 1 messages
[2018-08-21T16:51:00.520Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Repsonse status code 202
[2018-08-21T16:51:00.521Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Fired trigger with activation d76bfb31c36b4c22abfb31c36bec2243
  ➜  openwhisk master ✗   bin/wsk -i --apihost 172.17.0.1 --auth 789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP activation get d76bfb31c36b4c22abfb31c36bec2243
ok: got activation d76bfb31c36b4c22abfb31c36bec2243
{
    "namespace": "whisk.system",
    "name": "kafkatrigger",
    "version": "0.0.1",
    "subject": "whisk.system",
    "activationId": "d76bfb31c36b4c22abfb31c36bec2243",
    "start": 1534870260518,
    "end": 0,
    "duration": 0,
    "response": {
        "status": "success",
        "statusCode": 0,
        "success": true,
        "result": {
            "messages": [
                {
                    "key": null,
                    "offset": 44601,
                    "partition": 0,
                    "topic": "test",
                    "value": "\"Malalažbeta my weird character \\xb6\""
                }
            ]
        }
    },
    "logs": [
        "{\"statusCode\":0,\"success\":true,\"activationId\":\"7ba23bc128794a85a23bc128791a85b6\",\"rule\":\"whisk.system/myrule\",\"action\":\"whisk.system/myecho\"}"
    ],
    "annotations": [],
    "publish": false
}
>{"hello": "Malalažbeta my weird character \xb6"}
[2018-08-21T16:53:43.249Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Found 1 messages with a total size of 134 bytes
[2018-08-21T16:53:43.249Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Firing trigger with 1 messages
[2018-08-21T16:53:43.267Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Repsonse status code 202
[2018-08-21T16:53:43.267Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Fired trigger with activation a0897002581948948970025819589491
  ➜  openwhisk master ✗   bin/wsk -i --apihost 172.17.0.1 --auth 789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP activation get a0897002581948948970025819589491
ok: got activation a0897002581948948970025819589491
{
    "namespace": "whisk.system",
    "name": "kafkatrigger",
    "version": "0.0.1",
    "subject": "whisk.system",
    "activationId": "a0897002581948948970025819589491",
    "start": 1534870423265,
    "end": 0,
    "duration": 0,
    "response": {
        "status": "success",
        "statusCode": 0,
        "success": true,
        "result": {
            "messages": [
                {
                    "key": null,
                    "offset": 44602,
                    "partition": 0,
                    "topic": "test",
                    "value": "{\"hello\": \"Malalažbeta my weird character \\xb6\"}"
                }
            ]
        }
    },
    "logs": [
        "{\"statusCode\":0,\"success\":true,\"activationId\":\"64ef9022f17f4a49af9022f17f1a49e6\",\"rule\":\"whisk.system/myrule\",\"action\":\"whisk.system/myecho\"}"
    ],
    "annotations": [],
    "publish": false
}
>"{"hello": "Malalažbeta my weird character \xb6"}"
[2018-08-21T16:55:28.677Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Found 1 messages with a total size of 138 bytes
[2018-08-21T16:55:28.678Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Firing trigger with 1 messages
[2018-08-21T16:55:28.689Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Repsonse status code 202
[2018-08-21T16:55:28.689Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Fired trigger with activation eed5d571becf431195d571becfe3116e
  ➜  openwhisk master ✗   bin/wsk -i --apihost 172.17.0.1 --auth 789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP activation get eed5d571becf431195d571becfe3116e
ok: got activation eed5d571becf431195d571becfe3116e
{
    "namespace": "whisk.system",
    "name": "kafkatrigger",
    "version": "0.0.1",
    "subject": "whisk.system",
    "activationId": "eed5d571becf431195d571becfe3116e",
    "start": 1534870528687,
    "end": 0,
    "duration": 0,
    "response": {
        "status": "success",
        "statusCode": 0,
        "success": true,
        "result": {
            "messages": [
                {
                    "key": null,
                    "offset": 44603,
                    "partition": 0,
                    "topic": "test",
                    "value": "\"{\"hello\": \"Malalažbeta my weird character \\xb6\"}\""
                }
            ]
        }
    },
    "logs": [
        "{\"statusCode\":0,\"success\":true,\"activationId\":\"84bbce9c128b480bbbce9c128be80bbc\",\"rule\":\"whisk.system/myrule\",\"action\":\"whisk.system/myecho\"}"
    ],
    "annotations": [],
    "publish": false
}
>"{\"hello\": \"Malalažbeta my weird character \xb6\"}"
[2018-08-21T16:57:56.258Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Found 1 messages with a total size of 146 bytes
[2018-08-21T16:57:56.259Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Firing trigger with 1 messages
[2018-08-21T16:57:56.283Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Repsonse status code 202
[2018-08-21T16:57:56.284Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Fired trigger with activation a4d5fd5140f543e095fd5140f5e3e087
  ➜  openwhisk master ✗   bin/wsk -i --apihost 172.17.0.1 --auth 789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP activation get a4d5fd5140f543e095fd5140f5e3e087
ok: got activation a4d5fd5140f543e095fd5140f5e3e087
{
    "namespace": "whisk.system",
    "name": "kafkatrigger",
    "version": "0.0.1",
    "subject": "whisk.system",
    "activationId": "a4d5fd5140f543e095fd5140f5e3e087",
    "start": 1534870676281,
    "end": 0,
    "duration": 0,
    "response": {
        "status": "success",
        "statusCode": 0,
        "success": true,
        "result": {
            "messages": [
                {
                    "key": null,
                    "offset": 44604,
                    "partition": 0,
                    "topic": "test",
                    "value": "\"{\\\"hello\\\": \\\"Malalažbeta my weird character \\xb6\\\"}\""
                }
            ]
        }
    },
    "logs": [
        "{\"statusCode\":0,\"success\":true,\"activationId\":\"30c16a3120124617816a312012c61718\",\"rule\":\"whisk.system/myrule\",\"action\":\"whisk.system/myecho\"}"
    ],
    "annotations": [],
    "publish": false
}
>{\"hello\": \"Malalažbeta my weird character \xb6\"}
[2018-08-21T17:00:01.670Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Found 1 messages with a total size of 142 bytes
[2018-08-21T17:00:01.672Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Firing trigger with 1 messages
[2018-08-21T17:00:01.684Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Repsonse status code 202
[2018-08-21T17:00:01.684Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Fired trigger with activation 5276f991ecbd4189b6f991ecbdc189fe
  ➜  openwhisk master ✗   bin/wsk -i --apihost 172.17.0.1 --auth 789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP activation get 5276f991ecbd4189b6f991ecbdc189fe
ok: got activation 5276f991ecbd4189b6f991ecbdc189fe
{
    "namespace": "whisk.system",
    "name": "kafkatrigger",
    "version": "0.0.1",
    "subject": "whisk.system",
    "activationId": "5276f991ecbd4189b6f991ecbdc189fe",
    "start": 1534870801679,
    "end": 0,
    "duration": 0,
    "response": {
        "status": "success",
        "statusCode": 0,
        "success": true,
        "result": {
            "messages": [
                {
                    "key": null,
                    "offset": 44605,
                    "partition": 0,
                    "topic": "test",
                    "value": "{\\\"hello\\\": \\\"Malalažbeta my weird character \\xb6\\\"}"
                }
            ]
        }
    },
    "logs": [
        "{\"statusCode\":0,\"success\":true,\"activationId\":\"decb6c86ffa242148b6c86ffa2b2147f\",\"rule\":\"whisk.system/myrule\",\"action\":\"whisk.system/myecho\"}"
    ],
    "annotations": [],
    "publish": false
}
abaruni commented 6 years ago

Now, updating the trigger to isJSONData: true

  ➜  openwhisk master ✗   bin/wsk -i --apihost 172.17.0.1 --auth 789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP trigger update kafkatrigger -p isJSONData true
{
    "activationId": "dda0958786f747d9a0958786f747d920",
    "annotations": [
        {
            "key": "path",
            "value": "whisk.system/messaging/messageHubFeed"
        },
        {
            "key": "waitTime",
            "value": 64
        },
        {
            "key": "kind",
            "value": "nodejs:6"
        },
        {
            "key": "limits",
            "value": {
                "logs": 10,
                "memory": 256,
                "timeout": 60000
            }
        },
        {
            "key": "initTime",
            "value": 345
        }
    ],
    "duration": 1046,
    "end": 1534870963261,
    "logs": [],
    "name": "messageHubFeed",
    "namespace": "whisk.system",
    "publish": false,
    "response": {
        "result": {},
        "status": "success",
        "success": true
    },
    "start": 1534870962215,
    "subject": "whisk.system",
    "version": "0.0.1"
}
ok: updated trigger kafkatrigger
>Malalažbeta my weird character \xb6
[2018-08-21T17:03:42.363Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Found 1 messages with a total size of 121 bytes
[2018-08-21T17:03:42.363Z] [WARNING] [??] [kafkatriggers] [/whisk.system/kafkatrigger] I was asked to encode a message as JSON, but I failed.
[2018-08-21T17:03:42.363Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Firing trigger with 1 messages
[2018-08-21T17:03:42.381Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Repsonse status code 202
[2018-08-21T17:03:42.382Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Fired trigger with activation edb2a30fe7ae4f09b2a30fe7ae6f0982
  ➜  openwhisk master ✗   bin/wsk -i --apihost 172.17.0.1 --auth 789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP activation get edb2a30fe7ae4f09b2a30fe7ae6f0982
ok: got activation edb2a30fe7ae4f09b2a30fe7ae6f0982
{
    "namespace": "whisk.system",
    "name": "kafkatrigger",
    "version": "0.0.1",
    "subject": "whisk.system",
    "activationId": "edb2a30fe7ae4f09b2a30fe7ae6f0982",
    "start": 1534871022376,
    "end": 0,
    "duration": 0,
    "response": {
        "status": "success",
        "statusCode": 0,
        "success": true,
        "result": {
            "messages": [
                {
                    "key": null,
                    "offset": 44606,
                    "partition": 0,
                    "topic": "test",
                    "value": "\"Malalažbeta my weird character \\xb6\""
                }
            ]
        }
    },
    "logs": [
        "{\"statusCode\":0,\"success\":true,\"activationId\":\"20b9bdc81c4142a2b9bdc81c4172a273\",\"rule\":\"whisk.system/myrule\",\"action\":\"whisk.system/myecho\"}"
    ],
    "annotations": [],
    "publish": false
}
>"Malalažbeta my weird character \xb6"
[2018-08-21T17:05:18.194Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Found 1 messages with a total size of 125 bytes
[2018-08-21T17:05:18.194Z] [WARNING] [??] [kafkatriggers] [/whisk.system/kafkatrigger] I was asked to encode a message as JSON, but I failed.
[2018-08-21T17:05:18.194Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Firing trigger with 1 messages
[2018-08-21T17:05:18.204Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Repsonse status code 202
[2018-08-21T17:05:18.204Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Fired trigger with activation 18975da686bc4ad5975da686bcaad5ba
  ➜  openwhisk master ✗   bin/wsk -i --apihost 172.17.0.1 --auth 789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP activation get 18975da686bc4ad5975da686bcaad5ba
ok: got activation 18975da686bc4ad5975da686bcaad5ba
{
    "namespace": "whisk.system",
    "name": "kafkatrigger",
    "version": "0.0.1",
    "subject": "whisk.system",
    "activationId": "18975da686bc4ad5975da686bcaad5ba",
    "start": 1534871118201,
    "end": 0,
    "duration": 0,
    "response": {
        "status": "success",
        "statusCode": 0,
        "success": true,
        "result": {
            "messages": [
                {
                    "key": null,
                    "offset": 44607,
                    "partition": 0,
                    "topic": "test",
                    "value": "\"\"Malalažbeta my weird character \\xb6\"\""
                }
            ]
        }
    },
    "logs": [
        "{\"statusCode\":0,\"success\":true,\"activationId\":\"6a7eb5dfe438432dbeb5dfe438032d97\",\"rule\":\"whisk.system/myrule\",\"action\":\"whisk.system/myecho\"}"
    ],
    "annotations": [],
    "publish": false
}
>{"hello": "Malalažbeta my weird character \xb6"}
[2018-08-21T17:06:35.384Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Found 1 messages with a total size of 138 bytes
[2018-08-21T17:06:35.385Z] [WARNING] [??] [kafkatriggers] [/whisk.system/kafkatrigger] I was asked to encode a message as JSON, but I failed.
[2018-08-21T17:06:35.385Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Firing trigger with 1 messages
[2018-08-21T17:06:35.401Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Repsonse status code 202
[2018-08-21T17:06:35.401Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Fired trigger with activation 49385e19d5f944c5b85e19d5f914c5e2
ok: got activation 49385e19d5f944c5b85e19d5f914c5e2
{
    "namespace": "whisk.system",
    "name": "kafkatrigger",
    "version": "0.0.1",
    "subject": "whisk.system",
    "activationId": "49385e19d5f944c5b85e19d5f914c5e2",
    "start": 1534871195398,
    "end": 0,
    "duration": 0,
    "response": {
        "status": "success",
        "statusCode": 0,
        "success": true,
        "result": {
            "messages": [
                {
                    "key": null,
                    "offset": 44608,
                    "partition": 0,
                    "topic": "test",
                    "value": "\"{\"hello\": \"Malalažbeta my weird character \\xb6\"}\""
                }
            ]
        }
    },
    "logs": [
        "{\"statusCode\":0,\"success\":true,\"activationId\":\"0b5710a7c326437e9710a7c326e37ebf\",\"rule\":\"whisk.system/myrule\",\"action\":\"whisk.system/myecho\"}"
    ],
    "annotations": [],
    "publish": false
}
>"{"hello": "Malalažbeta my weird character \xb6"}"
[2018-08-21T17:08:15.841Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Found 1 messages with a total size of 142 bytes
[2018-08-21T17:08:15.847Z] [WARNING] [??] [kafkatriggers] [/whisk.system/kafkatrigger] I was asked to encode a message as JSON, but I failed.
[2018-08-21T17:08:15.848Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Firing trigger with 1 messages
[2018-08-21T17:08:15.893Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Repsonse status code 202
[2018-08-21T17:08:15.893Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Fired trigger with activation c09f85197c2648fd9f85197c2688fdba
  ➜  openwhisk master ✗   bin/wsk -i --apihost 172.17.0.1 --auth 789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP activation get c09f85197c2648fd9f85197c2688fdba
ok: got activation c09f85197c2648fd9f85197c2688fdba
{
    "namespace": "whisk.system",
    "name": "kafkatrigger",
    "version": "0.0.1",
    "subject": "whisk.system",
    "activationId": "c09f85197c2648fd9f85197c2688fdba",
    "start": 1534871295890,
    "end": 0,
    "duration": 0,
    "response": {
        "status": "success",
        "statusCode": 0,
        "success": true,
        "result": {
            "messages": [
                {
                    "key": null,
                    "offset": 44609,
                    "partition": 0,
                    "topic": "test",
                    "value": "\"\"{\"hello\": \"Malalažbeta my weird character \\xb6\"}\"\""
                }
            ]
        }
    },
    "logs": [
        "{\"statusCode\":0,\"success\":true,\"activationId\":\"e4a5533247c44451a5533247c41451f6\",\"rule\":\"whisk.system/myrule\",\"action\":\"whisk.system/myecho\"}"
    ],
    "annotations": [],
    "publish": false
}
>"{\"hello\": \"Malalažbeta my weird character \xb6\"}"
[2018-08-21T17:09:48.169Z] [WARNING] [??] [kafkatriggers] [/whisk.system/kafkatrigger] I was asked to encode a message as JSON, but I failed.
[2018-08-21T17:09:48.169Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Firing trigger with 1 messages
[2018-08-21T17:09:48.178Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Repsonse status code 202
[2018-08-21T17:09:48.178Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Fired trigger with activation c14b90087d2442408b90087d24a240f9
  ➜  openwhisk master ✗   bin/wsk -i --apihost 172.17.0.1 --auth 789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP activation get c14b90087d2442408b90087d24a240f9
ok: got activation c14b90087d2442408b90087d24a240f9
{
    "namespace": "whisk.system",
    "name": "kafkatrigger",
    "version": "0.0.1",
    "subject": "whisk.system",
    "activationId": "c14b90087d2442408b90087d24a240f9",
    "start": 1534871388176,
    "end": 0,
    "duration": 0,
    "response": {
        "status": "success",
        "statusCode": 0,
        "success": true,
        "result": {
            "messages": [
                {
                    "key": null,
                    "offset": 44610,
                    "partition": 0,
                    "topic": "test",
                    "value": "\"\"{\\\"hello\\\": \\\"Malalažbeta my weird character \\xb6\\\"}\"\""
                }
            ]
        }
    },
    "logs": [
        "{\"statusCode\":0,\"success\":true,\"activationId\":\"903d0d44e9d24f06bd0d44e9d26f06d4\",\"rule\":\"whisk.system/myrule\",\"action\":\"whisk.system/myecho\"}"
    ],
    "annotations": [],
    "publish": false
}
>{\"hello\": \"Malalažbeta my weird character \xb6\"}
[2018-08-21T17:11:47.930Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Found 1 messages with a total size of 146 bytes
[2018-08-21T17:11:47.930Z] [WARNING] [??] [kafkatriggers] [/whisk.system/kafkatrigger] I was asked to encode a message as JSON, but I failed.
[2018-08-21T17:11:47.930Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Firing trigger with 1 messages
[2018-08-21T17:11:47.946Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Repsonse status code 202
[2018-08-21T17:11:47.947Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Fired trigger with activation 5aaf6a7957704d0aaf6a795770ed0a4e
  ➜  openwhisk master ✗   bin/wsk -i --apihost 172.17.0.1 --auth 789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP activation get 5aaf6a7957704d0aaf6a795770ed0a4e
ok: got activation 5aaf6a7957704d0aaf6a795770ed0a4e
{
    "namespace": "whisk.system",
    "name": "kafkatrigger",
    "version": "0.0.1",
    "subject": "whisk.system",
    "activationId": "5aaf6a7957704d0aaf6a795770ed0a4e",
    "start": 1534871507939,
    "end": 0,
    "duration": 0,
    "response": {
        "status": "success",
        "statusCode": 0,
        "success": true,
        "result": {
            "messages": [
                {
                    "key": null,
                    "offset": 44611,
                    "partition": 0,
                    "topic": "test",
                    "value": "\"{\\\"hello\\\": \\\"Malalažbeta my weird character \\xb6\\\"}\""
                }
            ]
        }
    },
    "logs": [
        "{\"statusCode\":0,\"success\":true,\"activationId\":\"9b3583fea8e64cf3b583fea8e61cf3a4\",\"rule\":\"whisk.system/myrule\",\"action\":\"whisk.system/myecho\"}"
    ],
    "annotations": [],
    "publish": false
}
abaruni commented 6 years ago

eliminating the invalid character

{"hello": "Malalažbeta my weird character"}
[2018-08-21T17:13:49.009Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Found 1 messages with a total size of 122 bytes
[2018-08-21T17:13:49.010Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Firing trigger with 1 messages
[2018-08-21T17:13:49.047Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Repsonse status code 202
[2018-08-21T17:13:49.047Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Fired trigger with activation 06cbe359514a471a8be359514a071ae0
  ➜  openwhisk master ✗   bin/wsk -i --apihost 172.17.0.1 --auth 789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP activation get 06cbe359514a471a8be359514a071ae0
ok: got activation 06cbe359514a471a8be359514a071ae0
{
    "namespace": "whisk.system",
    "name": "kafkatrigger",
    "version": "0.0.1",
    "subject": "whisk.system",
    "activationId": "06cbe359514a471a8be359514a071ae0",
    "start": 1534871629042,
    "end": 0,
    "duration": 0,
    "response": {
        "status": "success",
        "statusCode": 0,
        "success": true,
        "result": {
            "messages": [
                {
                    "key": null,
                    "offset": 44612,
                    "partition": 0,
                    "topic": "test",
                    "value": {
                        "hello": "Malalažbeta my weird character"
                    }
                }
            ]
        }
    },
    "logs": [
        "{\"statusCode\":0,\"success\":true,\"activationId\":\"cdf959a0dcad486fb959a0dcad086fee\",\"rule\":\"whisk.system/myrule\",\"action\":\"whisk.system/myecho\"}"
    ],
    "annotations": [],
    "publish": false
}
abaruni commented 6 years ago

When encoding as json, if the encoding fails, then the data is returned as a string and the requests module can handle this fine when creating its json payload

when not encoded as json, again it appears as though python escapes this appropriately or the requests module escapes the invalid character appropriately when constructing its json payload

abaruni commented 6 years ago

remaining question i have is if we still need to do this 

logging.warn('[{}] I was asked to encode a message as JSON, but I failed.'.format(self.trigger))
value = "\"{}\"".format(value)

commenting out value = "\"{}\"".format(value)

>{"hello": "Malalažbeta my weird character \xb6"}
[2018-08-21T17:43:19.667Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Found 1 messages with a total size of 134 bytes
[2018-08-21T17:43:19.668Z] [WARNING] [??] [kafkatriggers] [/whisk.system/kafkatrigger] I was asked to encode a message as JSON, but I failed.
[2018-08-21T17:43:19.668Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Firing trigger with 1 messages
[2018-08-21T17:43:19.775Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Repsonse status code 202
[2018-08-21T17:43:19.776Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Fired trigger with activation 13700ae097b44f23b00ae097b41f2375
  ➜  openwhisk master ✗   bin/wsk -i --apihost 172.17.0.1 --auth 789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP activation get 13700ae097b44f23b00ae097b41f2375
ok: got activation 13700ae097b44f23b00ae097b41f2375
{
    "namespace": "whisk.system",
    "name": "kafkatrigger",
    "version": "0.0.1",
    "subject": "whisk.system",
    "activationId": "13700ae097b44f23b00ae097b41f2375",
    "start": 1534873399773,
    "end": 0,
    "duration": 0,
    "response": {
        "status": "success",
        "statusCode": 0,
        "success": true,
        "result": {
            "messages": [
                {
                    "key": null,
                    "offset": 44623,
                    "partition": 0,
                    "topic": "test",
                    "value": "{\"hello\": \"Malalažbeta my weird character \\xb6\"}"
                }
            ]
        }
    },
    "logs": [
        "{\"statusCode\":0,\"success\":true,\"activationId\":\"b5b86d55c39e437bb86d55c39e537b80\",\"rule\":\"whisk.system/myrule\",\"action\":\"whisk.system/myecho\"}"
    ],
    "annotations": [],
    "publish": false
}
>Malalažbeta my weird character
[2018-08-21T17:46:00.460Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Found 1 messages with a total size of 111 bytes
[2018-08-21T17:46:00.460Z] [WARNING] [??] [kafkatriggers] [/whisk.system/kafkatrigger] I was asked to encode a message as JSON, but I failed.
[2018-08-21T17:46:00.461Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Firing trigger with 1 messages
[2018-08-21T17:46:00.480Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Repsonse status code 202
[2018-08-21T17:46:00.480Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Fired trigger with activation a5f4ea6d4c8840cfb4ea6d4c88a0cfba
  ➜  openwhisk master ✗   bin/wsk -i --apihost 172.17.0.1 --auth 789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP activation get a5f4ea6d4c8840cfb4ea6d4c88a0cfba
ok: got activation a5f4ea6d4c8840cfb4ea6d4c88a0cfba
{
    "namespace": "whisk.system",
    "name": "kafkatrigger",
    "version": "0.0.1",
    "subject": "whisk.system",
    "activationId": "a5f4ea6d4c8840cfb4ea6d4c88a0cfba",
    "start": 1534873560476,
    "end": 0,
    "duration": 0,
    "response": {
        "status": "success",
        "statusCode": 0,
        "success": true,
        "result": {
            "messages": [
                {
                    "key": null,
                    "offset": 44624,
                    "partition": 0,
                    "topic": "test",
                    "value": "Malalažbeta my weird character"
                }
            ]
        }
    },
    "logs": [
        "{\"statusCode\":0,\"success\":true,\"activationId\":\"821fc59af01c41d39fc59af01c31d3cd\",\"rule\":\"whisk.system/myrule\",\"action\":\"whisk.system/myecho\"}"
    ],
    "annotations": [],
    "publish": false
}
>"Malalažbeta my weird character \xb6"
[2018-08-21T17:47:11.171Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Found 1 messages with a total size of 121 bytes
[2018-08-21T17:47:11.171Z] [WARNING] [??] [kafkatriggers] [/whisk.system/kafkatrigger] I was asked to encode a message as JSON, but I failed.
[2018-08-21T17:47:11.172Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Firing trigger with 1 messages
[2018-08-21T17:47:11.190Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Repsonse status code 202
[2018-08-21T17:47:11.190Z] [INFO] [??] [kafkatriggers] [/whisk.system/kafkatrigger] Fired trigger with activation 5d12c3b9e22c415692c3b9e22c5156b7
  ➜  openwhisk master ✗   bin/wsk -i --apihost 172.17.0.1 --auth 789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP activation get 5d12c3b9e22c415692c3b9e22c5156b7
ok: got activation 5d12c3b9e22c415692c3b9e22c5156b7
{
    "namespace": "whisk.system",
    "name": "kafkatrigger",
    "version": "0.0.1",
    "subject": "whisk.system",
    "activationId": "5d12c3b9e22c415692c3b9e22c5156b7",
    "start": 1534873631186,
    "end": 0,
    "duration": 0,
    "response": {
        "status": "success",
        "statusCode": 0,
        "success": true,
        "result": {
            "messages": [
                {
                    "key": null,
                    "offset": 44625,
                    "partition": 0,
                    "topic": "test",
                    "value": "\"Malalažbeta my weird character \\xb6\""
                }
            ]
        }
    },
    "logs": [
        "{\"statusCode\":0,\"success\":true,\"activationId\":\"7975e10335b94949b5e10335b9894942\",\"rule\":\"whisk.system/myrule\",\"action\":\"whisk.system/myecho\"}"
    ],
    "annotations": [],
    "publish": false
}

So, it looks like the input is already escaped as a string even without that line

abaruni commented 6 years ago

@dubee can you think of anything else needed for this?