Json in json - Githubissues

mwl commented 6 years ago
String encoded json inside json message field
{
  "app": {
    "name": "humio_nozzle",
    "id": "89019d17-194e-4366-8c29-a9a800cd20b7"
  },
  "instance": "pid-12@69db1dfd-2e99-487a-41db-1698",
  "org": {
    "name": "pcfdev-org",
    "id": "029d3ede-b878-4311-ab61-34674694d593"
  },
  "log": {
    "messagetype": "OUT",
    "sourcetype": "APP/PROC/WEB",
    "sourceinst": "1",
    "sourcetypekey": "APP/PROC/WEB-OUT",
    "message": "{\"timestamp\":\"1512134394.994653463\",\"source\":\"humio-nozzle\",\"message\":\"humio-nozzle.Humio returned an unexpected response: \\u003chtml\\u003e\\n  \\u003chead\\u003e\\n    \\u003ctitle\\u003eHumio is getting a new release\\u003c/title\\u003e\\n    \\u003cmeta http-equiv=\\\"refresh\\\" content=\\\"10\\\"\\u003e\\n  \\u003c/head\\u003e\\n  \\u003cbody bgcolor=\\\"#1F0035\\\"\\u003e\\n    \\u003ccenter\\u003e\\n      \\u003cbr\\u003e\\u003cbr\\u003e\\n      \\u003ch1 style=\\\"color:#E9ECF2\\\"\\u003eHumio is getting a new release\\u003c/h1\\u003e\\n      \\u003cIMG src=\\\"data:image/png;base64,\\n                iVBORw0KGgoAAAANSUhEUgAAASoAAABQCAYAAACu9Dg8AAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJ\\n                bWFnZVJlYWR5ccllPAAAAyppVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADw/eHBhY2tldCBiZWdp\\n                bj0i77u/IiBpZD0iVzVNME1wQ2VoaUh6cmVTek5UY3prYzlkIj8+IDx4OnhtcG1ldGEgeG1sbnM6\\n                eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IkFkb2JlIFhNUCBDb3JlIDUuNi1jMTMyIDc5LjE1\\n                OTI4NCwgMjAxNi8wNC8xOS0xMzoxMzo0MCAgICAgICAgIj4gPHJkZjpSREYgeG1sbnM6cmRmPSJo\\n                dHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4gPHJkZjpEZXNjcmlw\\n                dGlvbiByZGY6YWJvdXQ9IiIgeG1sbnM6eG1wTU09Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEu\\n                MC9tbS8iIHhtbG5zOnN0UmVmPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvc1R5cGUvUmVz\\n                b3VyY2VSZWYjIiB4bWxuczp4bXA9Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC8iIHhtcE1N\\n                OkRvY3VtZW50SUQ9InhtcC5kaWQ6NjUyRkQxM0I2Mzg4MTFFNkFFMDVGMDdEQkNFMTRCQjAiIHht\\n                cE1NOkluc3RhbmNlSUQ9InhtcC5paWQ6NjUyRkQxM0E2Mzg4MTFFNkFFMDVGMDdEQkNFMTRCQjAi\\n                IHhtcDpDcmVhdG9yVG9vbD0iQWRvYmUgUGhvdG9zaG9wIENDIDIwMTUuNSAoTWFjaW50b3NoKSI+\\n                IDx4bXBNTTpEZXJpdmVkRnJvbSBzdFJlZjppbnN0YW5jZUlEPSJ4bXAuaWlkOkE3NENCMDc5NjM4\\n                NTExRTZBRTA1RjA3REJDRTE0QkIwIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOkE3NENCMDdB\\n                NjM4NTExRTZBRTA1RjA3REJDRTE0QkIwIi8+IDwvcmRmOkRlc2NyaXB0aW9uPiA8L3JkZjpSREY+\\n                IDwveDp4bXBtZXRhPiA8P3hwYWNrZXQgZW5kPSJyIj8+5ngOwwAAEzBJREFUeNrsXQmUFcUV7QEE\\n                ARWQRRYVXEGMiDgqCEFEBfctijuux0TjHhO3RI1gQI1i3OIuokE9gAzEBRQQ1wBuLCqio4gCIusw\\n                DAIiTO7zv39O29TWf7r79//z7jn39EwvVdVV1e+/evXqVUl1dbUnEAgEaUYdqQKBQCCCSiAQCGqI\\n                ev5/SkpKEs0cw86dcDgI3BfsBO4CtgGbgw18t24CV4DLwPlgOTgLfB+ci3JvlqYUCIoXJX4bVdyC\\n                CnmR8OkHHgv2B9tHkOwqcAr4MliGd1glzSoQFBlIUGUZYx5dwYfAldXxYgM4FjwalGGtQCCCyint\\n                fuDk6vxgHngxWE9aWSAQQaVKsxScWp0OlIOngSXS2gKBCCpKoyn4KLi5On0gwdlJWlwgqMWCCs/3\\n                BxdVpxvrwWtEuxIIapmgwnN1waHVhYUJ4PbS+gJBYaBG7gl4tgkOo8AjCvDdyR/rWLzzZ9INBIIi\\n                FVR4bkccXgV/U8Dvvxo8Ae/9pnQFgaDIBBULqangbkVQB+tYs5oi3UEgSCdCO0VCSLVNSEhtZo3n\\n                p5jzaQi+hPfqI91BICgCjQr3bovD215mbV4VSPYdWq/X0gusGwyBDeA7LPxmg5+D36Es63z51seh\\n                NbgnDzV7gX3BZjV496XgEpCM6qQhVoI9ke8n0i2KC+g/zbjfdACbepm1oxXgQvBjtPmaiPKh76OL\\n                l1kaRvnU5Xy+Becgn5VFUp87eJm1ue35+9mWFQqSCcvBL4h437WJCypekjIePIZPzcL9XX3XW7HA\\n                asdH0rxocXETbrTsMfs3LSoeAY7zC6UQlUXl6Q2eBx4OrmUNrCJwXMFCabHvuAx5/szp3IvDlZws\\n                dahSXFsWQ+PSh9JCcWka8lsfQ35Uz10VlxYiv/JaIJzox2cgeCr/sOo6N30AH3uZSaEnwrY98qE2\\n                PZfzOdCSD/0QjwGHI5/vCqguG+NwAngkKwjtXB4DZ3qZdbhl4Lt45+qaFMLJPQHXbwlM8c8skg59\\n                b+C9JpHLRQz5lGlcJTrE9F59NPndW+QCqhX4CPhTDm4r68Bh4HYO+WwD3gVW5ZAPle1x1kzSXJed\\n                wcfANRG4BH0FXkf1FpugwrVe4KZaIqgIN4igKsi2HACuiuCj+hbsbcinN/hNBPlQWU9PYT3uDD4X\\n                0woTCkxwI5tzohNUON8InK/IsJgF1UawiwiqgmlDMmEMifiDIq3nbEVeZ+WorZkwJA2rJcicAl4L\\n                rk3A6Xou+FvXsrkYwG9jI2QuL74Vj2kPAfdgoxvZksjZkgzoE3OxTwXsVD3Bw9i4R3aZdWxrmga+\\n                gvRX55A01Qup5t0lKF9B4F/g5RGnSX33aRoOog+M4f52KttVow4hdL2XmX2+Kp9DZhye9cI7b9Nk\\n                WBV/M9uGqBv6Xmn97S043m61X5k0KpzryNpFdRiNitQ68C/gUotUrQD/wbMlYSX/RQ7q948cB6tV\\n                SI0qiwtEo0q9NnWVw6/3J+Ag8Hhwfyb9fSs4y2F96D7gfg6a1BzwNkU+fwdnO5TzsjzVYUce7rrE\\n                exsHXgn24AmboGa7M6/7vR380FG7Gm0dCloEVZkh8ZmGl54dUg1cAB7kWKltwDdDpr8CPCYHQfU9\\n                2FAEVWqFVDeL8KDhxZGOdWUSWDPBzyyC0DWfORZB0CUPdbjMwWZ3Gbt55GKQv48nKkyguHWNQgsq\\n                /H+AJeGZirS6Ory0Sfvpa3npncCvc0yfJgPOCimoCFeLoEqtXWq6od1GgFuHSK8+z8SFxXAOse2a\\n                TwPwSUN67yRlr0I+u1u+10rwitCGb72C8aylLv+rnXE3CKrRYQQVTwsvrqGBjSqmo6acDS2/Rq4G\\n                0l4hBdXiiBpKBFW073eCRXiU5Cj8HgzRn56oQT4moXhUAvXXjINK6vAW2C6GfClM+HJDvg85Cyr8\\n                vYvD1GRQUI2KaDbgf6rGZ5+VKDDfP5xzEFSEgSKoUieodBFkPw6j4SjSrWfR1LL4iCeLcs1nK3CG\\n                Ju3XEqg/kyLy7zhDeLMdy6R0DHAVVIMdGmpmiGFi1tA4jg1sNiF4cqCMOzoYMr9k1fE9wwTAFsM5\\n                R0H1tgiqVAmpnQ1t1SOC9Pd36BNx5kNmitYx1t/5hvcamlAbbg9+oCnDahoqGgUVq6UuMwAf+dJ4\\n                yrLJwoGBPDuB0wzPvB64/xaLwbufQrC9YhJqvnuHOWpiu4mgSo2gukjzblMizGOCoS8kkc/ZMQ75\\n                dLPxjyfcji3ZY12F//jvVal33b3MQmMbPs8KNi+zT58KP4CHlJSULPGfxP+f4zlan0cbiKrimB9K\\n                Lgu+xaLHadInn6m+uG9uIP2FeJ6emexlfLiCICNiJyqHl1lA6QLyoRnqFTE0Qqwc9fRAzPleiMM+\\n                ikt3Ie9FivM9NUmNjLBYz3mZvSfjzudZTT4H8zWlNoLDzYpLtPD5CUt+9FxLxXn6Fi9Nsr/Rukoe\\n                PU33fr3hMOFMXLsf90zTaVR3OGoY1/L97Q33XGHpoCcZnv2tz2agG8rdbUm/myH9s/me7o7vO73Y\\n                NSrN/VMTEJC6uumquf/dOLTeQB6mfr1HhPm0003XG57poHmmzJJXC83axHVR1l0OdaDzhRufvUfl\\n                RdrfMf3v+bij4R6bUXCC4Vp2xmEHT+9BP9EisWl4ulRzuS0fFzu+b6nEWU8N2ijObfQyKx6i+rVf\\n                gIMqqgWtVPg6wnwW8cggiJYx1NsfwcaK80NRjq/y2J6ksc9WnD8O39xeWwgq9jR1dTjbwEfTzIdt\\n                +Qp1hE2aa/Ud0q90KGeV5nyDwHvYQHXVS2REKqBagf9jDMudVP1rDfLZFHE+qxTntosyAzbRnKO4\\n                RGGQ7s5nY3LIpZs1lweqNCqyT7n6hTTn43LDPftZ0qAYTbqQKst9FZlT+uxJ296SfvMQddpdZEQq\\n                UD+hfNYllM/GBPKgvqsa3j0AQVGVgjalYZ5qo5UzVYIqjPt+1ghebqjoqy0OcSav77ksbcmgrgsy\\n                dpnFGfNygyD8lI97hXjnriIjBAUKnUlneBoKx4uSn1ZcIleUTkFBtXeItJtwBjR8e09zD83s3alx\\n                4KSomudrnvsG6frtDW9o7usMPqUSVjh3ikGdpAgOM3JQsTtLfxcUKPoozs3Ad/ZNisr4vOZ836CR\\n                ehfHBMn4daXv/2c8tRsAgWYHD6P1TSSAvIwhlGbcehvSH6H4X+cdTqohOZw+xloYDeV+5+ldGgij\\n                suF/cSRBt5GluS1EBfln1cuGMRYICgilinOvp6mA+K7If/MrxRC1NCioXNb2jAOvCMSPIUH1V08f\\n                t4psSfc7lrciey+FhEU+lSCtrCat7WDNMzRdfKdj+iRk7ghU0LPsCWtLg4aR5DG8MML2uYCiHsbQ\\n                7rvLtyng74hm5lWzfTNSWNzpCkG1Z1BQtbAkQlrIpcEgV/iflrf8wTO7G7jiz0hvOQfFe50cwngK\\n                9xKu2AY1TH8IO3oGcQ9rejY7XcuIBdXf5FMSxAydAjEvhWVVlWmP4FDHZq8ZiY98sUZtmxjBR/cY\\n                0sm68f/ey+zqcSenT34WF9cwfdrZ+TZN+WnKeZhDGk2k3wsKDDr/v+9TWFaVErBN2JCqL1rGmINx\\n                uNHLbJUTFvex1kSqKhn1s74d5Ep/LqdPtqrzvNw2JR0NnmyxL5Fnr4QeFhQbttF8r5UpLKvKVaJR\\n                WEFlHdPi5Yd4mdk+1zV0NKw7Bc9dSVoNu/KTduaPrElb9hzG6ZPRm3xC3ndMfyULwAG2/fNwnexj\\n                X0q/FhQZCunHV+nOFEZQrXfdnBH30epy0orIRWCst6XTJvlGTWDtaFdf8PxuOEzytjTqk3c6hXA5\\n                g9OnDSMpdDGFfx2pUGHJUe9NL+NH1R73Pxxi88MF0q8FRQalQye+pzSaMVTa39qgMZ1UQZ2dakOY\\n                3HiINYaZ9RKniqkiY7mi0mgDhqleZicLFUjDGkkzZGQPY8EzkZndTrs5C6mlNdiV1eaNvDrihiHh\\n                uySGBifn1GHyjQp4VKHCTjH055piZ8W5yqCgWu5FvMbIJ7hoPdMqw/WlHNFgtKdf33cHG+1Vz69h\\n                TS1uRL3d+7Q4nO5sO18LahV0C473BD9JWVlVK0W+DA79FhkS2C7O8KQsbMbzcHGjRkhdn0BFtTJc\\n                2xST9iMQxPld/eCpF1gfnMLiqso0LyioTGEyyMjVNoFKVQmrpISUZ3nHheKVLihQqOKpHZamAkIR\\n                Ig1P5XQ+IyioPrWklchat4CwSkxIoaLIkNfecMun0t8FBQpV+OSu2XhPKcEZmvNvBAXVbEtC3RJU\\n                V0lYdUtQk3J5v1nS3xOFaK/R4VXN+QvSUDg2K52ruPQFBfULCiqKT2yywh6a8Ng6aUNfb8v1aUXe\\n                mVUznklENdWF6qnyBFF9S/Qjq4r3dElwa/Y8gbbIUgVF+CVufJ3Ay1RYtKretFC4iNvzeMM1cpp7\\n                p8j7c4XiXNsE8tVtJrLSE0SJJxXnaLHyzXnWpmhH60GKSzR59cwWgoox0fLLd2QxtiDvCnuA4ZYP\\n                IMiL/cNRxQJvHseOub56p0Xmqp2IVqZ0iUch41FP7TdF27bvm8dy0bK7XRXnR2Vdd1SCaqwl0dOK\\n                tBEHWK6PrWH6a7mTBBnX8oafNfmZHFp1Q+3DY6x3ikOvcnuZY3hmtYZRozKP+VRatHtVudZahn/k\\n                Z6hyAqbwRS/wZFLSCgK1/02adxzsvzHsBqQ/g22LSULxO38e5wakBVIP52jefWKMeT6myXOwKECx\\n                1HcjcIGmzseCdRMsy67gEk1ZHgzenMuW7jcrMt3eEh89LQ21veJcn7i3dC+QTtyKtxNXYd8Y8msD\\n                /qjJTzbSiK+dTzT09aeSEFa8m3m5YffzZi6Cahdws+FllgfVRPw/Arw85Q1Ee/N9Am4VOD/BIqgG\\n                1qJO/JqmDt7iYIZR5jVck9f8QvjRK/B2fsTQ38eAjWPMe29woSZvkjv9VA9tIaj4/GjLx3t1QABQ\\n                Buvp75Q2TDPway77Vb7ztp2SF1t2uim2DnyMoS4GRZjPQJe+JYitnRuCHxva4FNwnxjyvRBca8j3\\n                Vt2DOkFVavmAl2f9L3Cc4jv/HcdoTlOj1Acn+cq4IqsR4viGq0CuJR2Y7HUzDPVxYwR5DAB/0qS/\\n                KB9G3VoqrFr7frxV2AgOjcLPirWoyZZv7SmtJq0TVHytzJLw3eD+ivNzwJYpaYy64EiVALKM1bNj\\n                5Ya1sAP3MNiqskOD1jmk2xgcZqnzM0WEJNrWu/NQ24QK8Ha6N2TadcDDwRctpiTC88agBxZB1ZGl\\n                qmkGcLJBddwxz42wNX9UKnzG2p8JF9TiDjzIUjdV4D0uwwPcQ5tI3sSC34SRIjry0tbtwFnVbviQ\\n                tayTWUtqSgKGh5I0GXMQeD7bHxc6pnm/zf5Z4hdQJSUlqpf4Jw5/yrEOKPLmiUh3Rh4qn37xKcZ7\\n                jxyToFDH3VH2zbW089LMzyjwJIfbacMPamNyGCVfHepU5G1MTnwUwG9PhzTo+UNR3z+K6MhLe5Px\\n                /GEvsxNTUiCfPtrVarhLAbUaFV9v5KAamrCBh1l1Eqz0Ix1+vastY/Mu0nl/+ZUcXx0/pqfFVCBt\\n                Xn0W+EMCbf422DlMwYyCiu/pZbFZ6LCJp7u/AF9IyD/jGnAZ+BK4OsdKvEG67K9sfPfE2GHJfthI\\n                ajpVbd6UbYnrYmjvcp7xLQlbKKug4vtuyaFQ1/m0siYJVTINZ3fgvw/OocyTkvTOLaDOewQ4L8IO\\n                u0gM56lv8xZsq/w2gvZ+hwVUvVwL4yqo6rCWEgYnWtIszVWAkdMmeIjDL0MYLJAhiLXOL+JZ3Vyx\\n                gLVecUEonHavw6s3yIj+vsG1xI9VLC+ujmL5mdWYHigw7fRCy0lcl1OchDTLDOnRtaM5zXe9jAG7\\n                nIyyeG5DwLBLoUDohff3Mvv6URjVatzX1CSoPMOGEgHQItCeeYiBVaidl2b7TvYyi4qpTZppbqWY\\n                UrO4fV+iY22doCiitietqANI7grbMEmQ0MJoijAyj+O0R5qpk0blu7+tYY1OWI2qzGLQrrCMkysi\\n                0qhovVkf6YI1tmt0Aruyb11n0U4FUSH0eBGScjF/1FNZw4mzbEnYtWiK9Fi811TpDjVQzTNBFyuk\\n                JgRxoE6OnXIhDiSsbMMk2xq5BjUs/1aW2QNb+qSqHsU7OwsEgmLRqPzCioNekVPgEZrbrqV4M15m\\n                9xoKYUyhfM8Bn/cy8ZH71rD8NK09lG1dp3uZAPbkuHaglwl0d6nh2fmsSX0m3UAgSL9tIZSNSvF8\\n                XZ4NKCRMUMWlEggERSqofOn0Z9+YNGM9T41LrCOBoDYKKk6LZn4edVgpnQ9MpVkpaXGBoJYLKl+a\\n                pSwY0gBypThNtCiBQASVLu1+DsGy4gIt97g4Z5d9gUBQOwSVLw9yAnwIXBmzcNrAO2kcnWS0BoFA\\n                EC9CLaGJQGCRXxMFbj8W7A+2jyBZWiJDflAvg2V4h1XSrAKBCKooBRet3zvIy6wdJEM3+Va1AZt7\\n                v3bWpK2dV4DLvIz/E60HpPVjtDZwrqwdEwhqkaASCASCNELsOAKBQASVQCAQ1BT/F2AA2fNEh62r\\n                9NQAAAAASUVORK5CYII=\\n                \\\"\\u003e\\n      \\u003ch2 style=\\\"color:#E9ECF2\\\"\\u003e...we will be up again in a minute\\u003c/h2\\u003e\\n    \\u003c/center\\u003e\\n  \\u003c/body\\u003e\\n\\u003c/html\\u003e\\n\",\"log_level\":2,\"data\":{}}",
    "timestamp": "2017-12-01T13:19:54Z"
  },
  "index": "0",
  "http": {
    "remoteaddr": "",
    "method": "",
    "starttimestamp": "",
    "ua": "",
    "uri": "",
    "forwarded": "",
    "statuscode": 0,
    "peertype": "",
    "instanceid": "",
    "requestid": "",
    "stoptimestamp": "",
    "instanceindex": 0,
    "contentlength": 0
  },
  "eventtype": "LogMessage",
  "env": "cf",
  "job": "pcfdev",
  "space": {
    "name": "pcfdev-space",
    "id": "771923d2-1bd5-48e7-a61b-252011b17851"
  },
  "timestamp": "2017-12-01T13:19:54Z",
  "deployment": "pcfdev"
}
Guess we could just inject the raw json on the message field?
mwl commented 6 years ago
Not to mention the, at least, three timestamps in there :)
russmiles commented 6 years ago
This is actually an oddity of some logging that I'm doing from the nozzle, and it's currently the only case where there would be any structured content. For this reason I suggest that these logging messages should be rare enough, and frankly uninteresting enough, to either be removed completely from the nozzle (not my preferred option as they are useful if you are observing the nozzle itself) or simply left as-is since taking on the overhead of trying to unpack these JSON-in_JSON oddities is likely not worth it given their niche nature.
mwl commented 6 years ago
Okay. I accept that. In the end we might implement the required query function to be able to parse the json content.
humio / cloudfoundry2humio

Json in json #6
