Gzure
commented
3 years ago @xieus Do we have more API GW logs for the request?
Closed xieus closed 3 years ago
Gzure
commented
3 years ago @xieus Do we have more API GW logs for the request?
xieus
commented
3 years ago @Gzure Let me paste a little more but not much from API GW logs:
2020-11-10 01:43:06.071 DEBUG 1 --- [or-http-epoll-1] reactor.netty.channel.FluxReceive : [id: 0x75b18893, L:/172.16.109.176:59198 - R:portmanager-service.default.svc.cluster.local/10.100.100.75:9006] Subscribing inbound receiver [pending: 0, cancelled:false, inboundDone: false]
2020-11-10 01:43:06.071 DEBUG 1 --- [or-http-epoll-1] r.n.http.client.HttpClientOperations : [id: 0x75b18893, L:/172.16.109.176:59198 - R:portmanager-service.default.svc.cluster.local/10.100.100.75:9006] Received last HTTP packet
2020-11-10 01:43:06.072 DEBUG 1 --- [or-http-epoll-1] r.n.resources.PooledConnectionProvider : [id: 0x75b18893, L:/172.16.109.176:59198 - R:portmanager-service.default.svc.cluster.local/10.100.100.75:9006] onStateChange(GET{uri=/project/0c664c24649c4605aba77246d4b0c63f/ports?device_id=2d2caba5-ee35-4c74-850f-19793481bdfb&device_id=620c173f-03ce-4285-ab67-c5ff214a37d9&device_id=a0168461-f67c-4a6a-b3c8-1045f1982f83&device_id=c674fa4f-6428-4ccf-968c-50f98c6e334e&device_id=0863e0cd-2e8b-4e9d-8479-f8a3d8ce53a6&device_id=0d25ad74-ce8c-4c33-8f68-9b3558fcec11&device_id=16e62417-e336-476a-8745-52266a4b278c&device_id=195975ee-c83c-4e43-b770-0b2dc0563956&device_id=45e1b11a-0cf8-4e0c-94d6-6cadce49d548&device_id=542207cd-27f3-4523-88bb-6dfc2f2aaa2e&device_id=5bbc2ed3-3513-4dba-99ed-1a6f4a47df2b&device_id=5f4bf89b-3d83-4426-bcfc-9829490a22cf&device_id=6db2adf1-f600-4f5e-a0be-17619e1bdae7&device_id=828e0aad-ec1c-4ce8-b979-51669dcf7fb7&device_id=8532d38f-9e57-4e15-a0db-80c54f7820ac&device_id=91884ae0-f6ce-4b0c-990c-8426502d961e&device_id=95fd1ca3-cafe-423f-8a70-65a01fb4c033&device_id=a3418f5d-bcfb-4b8a-a6b9-acb886cff858&device_id=b4a0ad16-92f4-4a17-acfe-4a4ba915293d&device_id=c2cbccf9-b5aa-4b3f-9ac0-a5d9394c93ec&device_id=c47c37e6-a96e-4028-9af1-6c48bcb486cc&device_id=cabcc7c2-5f10-4418-b4dd-5dd6c50b02c5&device_id=edf856d1-df23-4d74-8350-508c071b35b5&device_id=f1247529-a762-4139-abbc-8246278bed21&device_id=f47aa0dd-3c68-4367-adb9-9c86474fb746&device_id=fa4538fa-1010-4c64-989e-a6afe2c4f425&device_id=fc6b79d8-fc84-40e8-89ac-72eee64a0d65&device_id=108858d2-f9b5-47a5-8999-2e36b4903b16&device_id=133e62cd-0ccc-4581-b767-88c5729e31fb&device_id=23ad6e3f-60b8-46b1-95e2-d2634072b09a&device_id=27848d59-389e-4f05-8abb-ef94d42e1128&device_id=2807af46-e128-4419-abe9-0ed7fe6b4556&device_id=2fb2d76b-e6d2-4ffe-94ac-e161709ad9b5&device_id=4f03669b-0915-442c-a3ec-d7755400203b&device_id=560d847a-b0cb-40bf-aa8d-4119f88190ca&device_id=5af14455-8c8f-4f63-a2ff-e39a2d60466d&device_id=633ccc71-b33a-489d-ac52-413719b7c91e&device_id=745bdb22-df1f-4d30-a8cc-6c5d6e48b4fb&device_id=74ee0a00-155a-49cd-8000-098e1b0630cb&device_id=77e28984-7055-4ce2-a8f5-cc064ec27d47&device_id=7a374323-ff11-46cc-8425-cbe881b448ef&device_id=8fee06fa-3f30-415f-9a92-cf14cb452625&device_id=92b3626c-845b-4a1c-9eef-852382b9cd29&device_id=92d6c2e6-fc04-47c2-aea4-41ac2924a9ec&device_id=93ebc33a-9a2c-41ec-b1f2-b56d31a0ed26&device_id=9e84dd65-56f9-4a55-a931-166a095de62a&device_id=a0bd2545-9e7e-46c2-a450-0a96fbbc532e&device_id=a39d4130-99f5-4130-b5e1-5ba98ce12c2a&device_id=a3c73407-811b-4c4d-966d-e12e870c1582&device_id=abc46ccd-82ae-4202-9f1e-fdc8b2bcbb19&device_id=bd997f95-28fd-4817-a7a0-da09a34af305&device_id=be888734-e953-4f04-b801-1597cdacaad2&device_id=c6d089df-a662-4083-af1c-2db3de3d74de&device_id=d7b8b420-c49a-4715-843e-b515b1fef975&device_id=da9e8e0b-d379-4551-ada9-2cc4a3c05c42&device_id=e28e6892-1710-495a-87df-972b39267c27&device_id=ed46a188-76d8-4149-bdcb-d1271c659111&device_id=ef7ccb72-a15f-4683-b3dc-060fcb927842&device_id=f8b538b9-8744-4045-8d7c-a666a163a81f&device_id=fce7479b-79ef-49cc-a902-4d29ad0fec89&device_id=157f9316-9776-4b7f-ac8d-ad7b92bd8831&device_id=171ece2b-8b43-4cde-b966-5133e9e6ce2b&device_id=2720fd9a-ed9b-4788-9d82-27bff7df9cdc&device_id=340f1a6b-3625-45f5-8fba-5e22e1cc3f78&device_id=3d316c2c-50a9-4883-b850-a083e2f30f5b&device_id=406ff289-d63d-4f70-8bb6-af3631308980&device_id=40fc9d52-92f4-43ca-87bd-3c0a93a3fcbf&device_id=47f6d477-37ff-40d2-9e13-c2a2227f4f70&device_id=53706fd0-06bc-4f76-a5fc-e925cea99df7&device_id=5e546575-dc12-4013-b630-961b64d5a369&device_id=66163c5b-e338-48c9-ad33-86b43fa51ab9&device_id=681d0727-15c5-4fe7-a809-c78012a090c4&device_id=6c1688fe-19bb-49b1-ae61-1b63687f7887&device_id=8829c3ca-e49b-474f-b034-0380d124c292&device_id=90776e20-eff6-42b2-9560-a57f4727f724&device_id=ad4f1fba-0003-438a-8fab-55ee9dd59a83&device_id=b021d095-9144-406f-90f6-77da4c449671&device_id=b11290ae-763b-49f4-a6cd-a72cc1de79a8&device_id=b3d377d9-9440-4e13-b941-fb76c3024ee3&device_id=cbe4e7d3-e4d4-4225-bfbe-192d72ad854d&device_id=d9b4cd8e-1023-4f64-9fe1-25e1f8f23d32&device_id=da9802de-3dbc-40ec-97a1-b973d34b66f1&device_id=e6d53118-edab-465c-9d89-17aa44f774bf&device_id=e772a9d2-f731-4ce6-a68e-79523ac72dfb&device_id=ec969faa-eb5d-45ce-b736-c29de21784c3&device_id=ef43665f-4b9e-464c-85a8-f16cfb93d082&device_id=15381f9d-7e9e-4d0c-b995-4423ab27ebde&device_id=24badaf2-07e4-4400-b595-bb7a02ec0b72&device_id=2ab2c5e7-08a7-44b5-a1fe-cd6d0f12bb58&device_id=2f58c899-8527-448e-91e5-d689b7e464a9&device_id=315a22f9-ea01-4697-97a0-294493542f31&device_id=41c6908b-c25b-4ec2-92c2-149e910b8a41&device_id=733ab157-4841-4a27-9dcb-751c29ac97be&device_id=748cbe7d-e66c-4d26-ac0f-ff95cbb6c9dd&device_id=8782704f-54c0-44c3-b04a-ca6c42f22dc0&device_id=b544a79f-0852-4fe5-8a8a-53254fb5e9f1&device_id=bc8db389-29c0-40f7-aafd-96f2bdd77859&device_id=e420641f-7c07-498c-a6d6-47fec7f1a933&device_id=eaea5d2b-275b-4513-9493-36605b34fb1b&device_id=ee0e47e4-66dc-4cee-ae1d-5df483e32f45&device_id=02f4b757-2536-4ea7-8700-74594cb5de57&device_id=03dd5d38-52a9-45af-aaae-0bc235e6f1ff&device_id=04658867-e7d4-4564-9b8d-25b520fabda7&device_id=143d80ac-0d4a-498e-8e9a-80d865483c6f&device_id=28b62b76-af05-4115-9c06-3feb14c18e33&device_id=2b99b7a0-5a31-4a97-a786-84eede30e638&device_id=2c2b1e76-2bb5-4e5e-a8c5-37fa2ec710df&device_id=2f58387a-4295-48f2-935f-6b097af50740&device_id=368e8a70-b586-4e67-83bf-9b51a7fc39dd&device_id=36c49870-58c6-42ec-874b-bd0ab9302e86&device_id=3f72b077-069a-4c5b-8f29-0c48b5c15e0f&device_id=4594aab7-61c2-4233-b27a-3e390e0118b2&device_id=480f1180-8e7a-48e4-85d2-4194dc82ec55&device_id=5f390f3e-6992-45a5-967d-68ce73acbc3b&device_id=63012149-ae4f-4713-b4f3-d48f53efeebd&device_id=7190d587-af64-4d53-a0e3-c201edc384ef&device_id=71e28f9e-04f2-4431-a183-e619eddbe559&device_id=874eb6b8-fce6-4c19-a402-026eb93c0399&device_id=8b6fb21a-ecca-4a32-a93c-144c0c86196a&device_id=97fe02a1-8116-4f97-8926-193bad57be05&device_id=9958306f-2dd1-4835-a4d1-dc1cf19a890c&device_id=997d7701-f769-4df9-8029-164b808a7ed1&device_id=a11b5619-bdd5-487c-9306-dd037529cf12&device_id=a880a472-2705-4d31-948c-0a9d0fd1e067&device_id=afa9834a-fea3-45f1-a5b1-7b1e4464f477&device_id=b3cae8ce-cc30-4ce5-b17d-6a6184aebaf4&device_id=b64459b1-2163-4b76-82ee-16bf7d35619f&device_id=b8187e1b-f862-4be3-89ed-49ebf667abc7&device_id=cbf2de40-9ec4-459d-82a3-9feafee72b47&device_id=d0d6e514-796c-4a97-a9b5-e915351ae83d&device_id=d65932bf-2123-4d41-a58f-0dbe29d5415e&device_id=d8a2ebd9-1967-4cba-8394-8ff96da0327a&device_id=db311029-9dc4-46bd-bc3b-c7dc6e35725c&device_id=dd0104c4-d6ae-41ac-b403-2a08742abbca&device_id=e8f20d40-43e2-49c4-8ad3-3498e9d2bff0&device_id=eb6c06ad-6b12-4fa8-87c4-0ea5fe43ab6f&device_id=ed9832ef-9d59-416f-96eb-4d9774d4ae20&device_id=f519fb0a-67d1-4eb4-bb62-0fe34c8e71c4&device_id=f90b94f7-72b2-47e1-883a-ed465c7aa9b3&device_id=feb458c0-d9b4-4c41-9431-aa6dd3763780&device_id=14b05793-76d6-4635-8fce-4443bf57fb38&device_id=55945312-e118-4f6c-9ecb-48660bf5aa97&device_id=7f3ff34c-c8c1-4a6f-a492-5ff80974404c&device_id=925c3be2-8b6b-4f10-816f-58075adfee03&device_id=bd4b9147-7522-48aa-b7ff-66b61be1ffd6&device_id=d4f56118-ac93-44dd-93b1-5d89d1c3e9f7&device_id=d5c5eb14-2597-4cae-a0e3-2bfae6510bb6&device_id=e8d3255f-b44a-4c75-8b2b-8844f4e01d79&device_id=eb2a36dd-12d3-42bf-a2bf-ba15b8c1782f&device_id=fde6d9fd-beba-4deb-aaf4-1526ed304d59, connection=PooledConnection{channel=[id: 0x75b18893, L:/172.16.109.176:59198 - R:portmanager-service.default.svc.cluster.local/10.100.100.75:9006]}}, [disconnecting])
2020-11-10 01:43:06.072 DEBUG 1 --- [or-http-epoll-1] r.n.resources.PooledConnectionProvider : [id: 0x75b18893, L:/172.16.109.176:59198 ! R:portmanager-service.default.svc.cluster.local/10.100.100.75:9006] Channel cleaned, now 0 active connections and 1 inactive connections
2020-11-10 01:43:06.072 DEBUG 1 --- [or-http-epoll-1] r.n.resources.PooledConnectionProvider : [id: 0x75b18893, L:/172.16.109.176:59198 ! R:portmanager-service.default.svc.cluster.local/10.100.100.75:9006] Channel closed, now 0 active connections and 0 inactive connections
2020-11-10 01:43:06.072 DEBUG 1 --- [or-http-epoll-1] r.n.resources.PooledConnectionProvider : [id: 0x75b18893, L:/172.16.109.176:59198 ! R:portmanager-service.default.svc.cluster.local/10.100.100.75:9006] onStateChange(PooledConnection{channel=[id: 0x75b18893, L:/172.16.109.176:59198 ! R:portmanager-service.default.svc.cluster.local/10.100.100.75:9006]}, [disconnecting])
2020-11-10 01:43:06.072 DEBUG 1 --- [or-http-epoll-3] o.s.w.s.adapter.HttpWebHandlerAdapter : [bf44d308] Completed 400 BAD_REQUEST
2020-11-10 01:43:06.072 DEBUG 1 --- [or-http-epoll-3] r.n.http.server.HttpServerOperations : [id: 0xbf44d308, L:/172.16.109.176:8080 - R:/10.213.43.170:50802] Last HTTP response frame
2020-11-10 01:43:06.072 DEBUG 1 --- [or-http-epoll-3] r.n.http.server.HttpServerOperations : [id: 0xbf44d308, L:/172.16.109.176:8080 - R:/10.213.43.170:50802] Decreasing pending responses, now 0
2020-11-10 01:43:06.072 DEBUG 1 --- [or-http-epoll-3] r.n.http.server.HttpServerOperations : [id: 0xbf44d308, L:/172.16.109.176:8080 - R:/10.213.43.170:50802] Last HTTP packet was sent, terminating the channelanother one:
2020-11-10 04:01:27.563 DEBUG 1 --- [or-http-epoll-4] reactor.netty.channel.FluxReceive : [id: 0x6a64df0a, L:/172.16.109.176:48820 - R:portmanager-service.default.svc.cluster.local/10.100.100.75:9006] Subscribing inbound receiver [pending: 0, cancelled:false, inboundDone: false]
2020-11-10 04:01:27.563 DEBUG 1 --- [or-http-epoll-4] r.n.http.client.HttpClientOperations : [id: 0x6a64df0a, L:/172.16.109.176:48820 - R:portmanager-service.default.svc.cluster.local/10.100.100.75:9006] Received last HTTP packet
2020-11-10 04:01:27.564 DEBUG 1 --- [or-http-epoll-4] r.n.resources.PooledConnectionProvider : [id: 0x6a64df0a, L:/172.16.109.176:48820 - R:portmanager-service.default.svc.cluster.local/10.100.100.75:9006] onStateChange(GET{uri=/project/f6f8eb223c114f0cb3aa965328c7f1a6/ports?device_id=2d2caba5-ee35-4c74-850f-19793481bdfb&device_id=a0168461-f67c-4a6a-b3c8-1045f1982f83&device_id=c674fa4f-6428-4ccf-968c-50f98c6e334e&device_id=620c173f-03ce-4285-ab67-c5ff214a37d9&device_id=f1247529-a762-4139-abbc-8246278bed21&device_id=0d25ad74-ce8c-4c33-8f68-9b3558fcec11&device_id=f47aa0dd-3c68-4367-adb9-9c86474fb746&device_id=91884ae0-f6ce-4b0c-990c-8426502d961e&device_id=fa4538fa-1010-4c64-989e-a6afe2c4f425&device_id=a3418f5d-bcfb-4b8a-a6b9-acb886cff858&device_id=542207cd-27f3-4523-88bb-6dfc2f2aaa2e&device_id=195975ee-c83c-4e43-b770-0b2dc0563956&device_id=c47c37e6-a96e-4028-9af1-6c48bcb486cc&device_id=45e1b11a-0cf8-4e0c-94d6-6cadce49d548&device_id=828e0aad-ec1c-4ce8-b979-51669dcf7fb7&device_id=cabcc7c2-5f10-4418-b4dd-5dd6c50b02c5&device_id=6db2adf1-f600-4f5e-a0be-17619e1bdae7&device_id=b4a0ad16-92f4-4a17-acfe-4a4ba915293d&device_id=fc6b79d8-fc84-40e8-89ac-72eee64a0d65&device_id=c2cbccf9-b5aa-4b3f-9ac0-a5d9394c93ec&device_id=95fd1ca3-cafe-423f-8a70-65a01fb4c033&device_id=5bbc2ed3-3513-4dba-99ed-1a6f4a47df2b&device_id=edf856d1-df23-4d74-8350-508c071b35b5&device_id=5f4bf89b-3d83-4426-bcfc-9829490a22cf&device_id=16e62417-e336-476a-8745-52266a4b278c&device_id=0863e0cd-2e8b-4e9d-8479-f8a3d8ce53a6&device_id=8532d38f-9e57-4e15-a0db-80c54f7820ac&device_id=f8b538b9-8744-4045-8d7c-a666a163a81f&device_id=2807af46-e128-4419-abe9-0ed7fe6b4556&device_id=77e28984-7055-4ce2-a8f5-cc064ec27d47&device_id=93ebc33a-9a2c-41ec-b1f2-b56d31a0ed26&device_id=a39d4130-99f5-4130-b5e1-5ba98ce12c2a&device_id=108858d2-f9b5-47a5-8999-2e36b4903b16&device_id=bd997f95-28fd-4817-a7a0-da09a34af305&device_id=fce7479b-79ef-49cc-a902-4d29ad0fec89&device_id=da9e8e0b-d379-4551-ada9-2cc4a3c05c42&device_id=74ee0a00-155a-49cd-8000-098e1b0630cb&device_id=4f03669b-0915-442c-a3ec-d7755400203b&device_id=e28e6892-1710-495a-87df-972b39267c27&device_id=5af14455-8c8f-4f63-a2ff-e39a2d60466d&device_id=7a374323-ff11-46cc-8425-cbe881b448ef&device_id=8fee06fa-3f30-415f-9a92-cf14cb452625&device_id=92d6c2e6-fc04-47c2-aea4-41ac2924a9ec&device_id=d7b8b420-c49a-4715-843e-b515b1fef975&device_id=92b3626c-845b-4a1c-9eef-852382b9cd29&device_id=27848d59-389e-4f05-8abb-ef94d42e1128&device_id=2fb2d76b-e6d2-4ffe-94ac-e161709ad9b5&device_id=ed46a188-76d8-4149-bdcb-d1271c659111&device_id=ef7ccb72-a15f-4683-b3dc-060fcb927842&device_id=633ccc71-b33a-489d-ac52-413719b7c91e&device_id=560d847a-b0cb-40bf-aa8d-4119f88190ca&device_id=c6d089df-a662-4083-af1c-2db3de3d74de&device_id=a3c73407-811b-4c4d-966d-e12e870c1582&device_id=745bdb22-df1f-4d30-a8cc-6c5d6e48b4fb&device_id=133e62cd-0ccc-4581-b767-88c5729e31fb&device_id=a0bd2545-9e7e-46c2-a450-0a96fbbc532e&device_id=abc46ccd-82ae-4202-9f1e-fdc8b2bcbb19&device_id=9e84dd65-56f9-4a55-a931-166a095de62a&device_id=be888734-e953-4f04-b801-1597cdacaad2&device_id=23ad6e3f-60b8-46b1-95e2-d2634072b09a&device_id=b021d095-9144-406f-90f6-77da4c449671&device_id=66163c5b-e338-48c9-ad33-86b43fa51ab9&device_id=5e546575-dc12-4013-b630-961b64d5a369&device_id=e772a9d2-f731-4ce6-a68e-79523ac72dfb&device_id=171ece2b-8b43-4cde-b966-5133e9e6ce2b&device_id=b3d377d9-9440-4e13-b941-fb76c3024ee3&device_id=53706fd0-06bc-4f76-a5fc-e925cea99df7&device_id=2720fd9a-ed9b-4788-9d82-27bff7df9cdc&device_id=406ff289-d63d-4f70-8bb6-af3631308980&device_id=47f6d477-37ff-40d2-9e13-c2a2227f4f70&device_id=d9b4cd8e-1023-4f64-9fe1-25e1f8f23d32&device_id=cbe4e7d3-e4d4-4225-bfbe-192d72ad854d&device_id=157f9316-9776-4b7f-ac8d-ad7b92bd8831&device_id=da9802de-3dbc-40ec-97a1-b973d34b66f1&device_id=6c1688fe-19bb-49b1-ae61-1b63687f7887&device_id=8829c3ca-e49b-474f-b034-0380d124c292&device_id=ef43665f-4b9e-464c-85a8-f16cfb93d082&device_id=ec969faa-eb5d-45ce-b736-c29de21784c3&device_id=340f1a6b-3625-45f5-8fba-5e22e1cc3f78&device_id=40fc9d52-92f4-43ca-87bd-3c0a93a3fcbf&device_id=90776e20-eff6-42b2-9560-a57f4727f724&device_id=681d0727-15c5-4fe7-a809-c78012a090c4&device_id=ad4f1fba-0003-438a-8fab-55ee9dd59a83&device_id=3d316c2c-50a9-4883-b850-a083e2f30f5b&device_id=e6d53118-edab-465c-9d89-17aa44f774bf&device_id=ee0e47e4-66dc-4cee-ae1d-5df483e32f45&device_id=748cbe7d-e66c-4d26-ac0f-ff95cbb6c9dd&device_id=24badaf2-07e4-4400-b595-bb7a02ec0b72&device_id=b544a79f-0852-4fe5-8a8a-53254fb5e9f1&device_id=15381f9d-7e9e-4d0c-b995-4423ab27ebde&device_id=315a22f9-ea01-4697-97a0-294493542f31&device_id=8782704f-54c0-44c3-b04a-ca6c42f22dc0&device_id=2ab2c5e7-08a7-44b5-a1fe-cd6d0f12bb58&device_id=bc8db389-29c0-40f7-aafd-96f2bdd77859&device_id=2f58c899-8527-448e-91e5-d689b7e464a9&device_id=eaea5d2b-275b-4513-9493-36605b34fb1b&device_id=e420641f-7c07-498c-a6d6-47fec7f1a933&device_id=733ab157-4841-4a27-9dcb-751c29ac97be&device_id=41c6908b-c25b-4ec2-92c2-149e910b8a41&device_id=b8187e1b-f862-4be3-89ed-49ebf667abc7&device_id=36c49870-58c6-42ec-874b-bd0ab9302e86&device_id=03dd5d38-52a9-45af-aaae-0bc235e6f1ff&device_id=b64459b1-2163-4b76-82ee-16bf7d35619f&device_id=97fe02a1-8116-4f97-8926-193bad57be05&device_id=143d80ac-0d4a-498e-8e9a-80d865483c6f&device_id=7190d587-af64-4d53-a0e3-c201edc384ef&device_id=f519fb0a-67d1-4eb4-bb62-0fe34c8e71c4&device_id=5f390f3e-6992-45a5-967d-68ce73acbc3b&device_id=71e28f9e-04f2-4431-a183-e619eddbe559&device_id=a11b5619-bdd5-487c-9306-dd037529cf12&device_id=cbf2de40-9ec4-459d-82a3-9feafee72b47&device_id=a880a472-2705-4d31-948c-0a9d0fd1e067&device_id=2c2b1e76-2bb5-4e5e-a8c5-37fa2ec710df&device_id=2f58387a-4295-48f2-935f-6b097af50740&device_id=feb458c0-d9b4-4c41-9431-aa6dd3763780&device_id=368e8a70-b586-4e67-83bf-9b51a7fc39dd&device_id=d8a2ebd9-1967-4cba-8394-8ff96da0327a&device_id=4594aab7-61c2-4233-b27a-3e390e0118b2&device_id=d65932bf-2123-4d41-a58f-0dbe29d5415e&device_id=e8f20d40-43e2-49c4-8ad3-3498e9d2bff0&device_id=02f4b757-2536-4ea7-8700-74594cb5de57&device_id=63012149-ae4f-4713-b4f3-d48f53efeebd&device_id=480f1180-8e7a-48e4-85d2-4194dc82ec55&device_id=8b6fb21a-ecca-4a32-a93c-144c0c86196a&device_id=eb6c06ad-6b12-4fa8-87c4-0ea5fe43ab6f&device_id=db311029-9dc4-46bd-bc3b-c7dc6e35725c&device_id=b3cae8ce-cc30-4ce5-b17d-6a6184aebaf4&device_id=28b62b76-af05-4115-9c06-3feb14c18e33&device_id=f90b94f7-72b2-47e1-883a-ed465c7aa9b3&device_id=874eb6b8-fce6-4c19-a402-026eb93c0399&device_id=2b99b7a0-5a31-4a97-a786-84eede30e638&device_id=04658867-e7d4-4564-9b8d-25b520fabda7&device_id=ed9832ef-9d59-416f-96eb-4d9774d4ae20&device_id=3f72b077-069a-4c5b-8f29-0c48b5c15e0f&device_id=dd0104c4-d6ae-41ac-b403-2a08742abbca&device_id=9958306f-2dd1-4835-a4d1-dc1cf19a890c&device_id=d0d6e514-796c-4a97-a9b5-e915351ae83d&device_id=afa9834a-fea3-45f1-a5b1-7b1e4464f477&device_id=997d7701-f769-4df9-8029-164b808a7ed1&device_id=b617e04b-b08b-4408-9541-9047d410d10c&device_id=ca2d692d-759c-4184-a7bd-ef83465f64e6&device_id=79824d01-2cb1-4b05-b881-2cf674ca2df4&device_id=2a1f8813-45e3-48ad-a4cb-c372f3153cde&device_id=6833cda4-3684-44a3-bfac-cf1a2bf9f99d&device_id=00529586-6b29-468b-845f-99d8e340552c&device_id=da93586d-521c-46b9-bc54-46ca5692b349&device_id=196196e2-2f1f-41fb-b5fc-da5171967c1f&device_id=d655f018-6573-4eff-ba2d-312143e71370&device_id=df0b6b2d-4b73-4486-b897-9ce29d15fe43&device_id=f63fb2ad-0158-40b2-9a9a-f68e984c2fff, connection=PooledConnection{channel=[id: 0x6a64df0a, L:/172.16.109.176:48820 - R:portmanager-service.default.svc.cluster.local/10.100.100.75:9006]}}, [disconnecting])
2020-11-10 04:01:27.564 DEBUG 1 --- [or-http-epoll-4] r.n.resources.PooledConnectionProvider : [id: 0x6a64df0a, L:/172.16.109.176:48820 ! R:portmanager-service.default.svc.cluster.local/10.100.100.75:9006] Channel cleaned, now 0 active connections and 1 inactive connections
2020-11-10 04:01:27.564 DEBUG 1 --- [or-http-epoll-4] r.n.resources.PooledConnectionProvider : [id: 0x6a64df0a, L:/172.16.109.176:48820 ! R:portmanager-service.default.svc.cluster.local/10.100.100.75:9006] Channel closed, now 0 active connections and 0 inactive connections
2020-11-10 04:01:27.564 DEBUG 1 --- [or-http-epoll-4] r.n.resources.PooledConnectionProvider : [id: 0x6a64df0a, L:/172.16.109.176:48820 ! R:portmanager-service.default.svc.cluster.local/10.100.100.75:9006] onStateChange(PooledConnection{channel=[id: 0x6a64df0a, L:/172.16.109.176:48820 ! R:portmanager-service.default.svc.cluster.local/10.100.100.75:9006]}, [disconnecting])
2020-11-10 04:01:27.564 DEBUG 1 --- [or-http-epoll-3] o.s.w.s.adapter.HttpWebHandlerAdapter : [0473af0b] Completed 400 BAD_REQUEST
2020-11-10 04:01:27.564 DEBUG 1 --- [or-http-epoll-3] r.n.http.server.HttpServerOperations : [id: 0x0473af0b, L:/172.16.109.176:8080 - R:/172.16.109.176:53656] Last HTTP response frame
2020-11-10 04:01:27.564 DEBUG 1 --- [or-http-epoll-3] r.n.http.server.HttpServerOperations : [id: 0x0473af0b, L:/172.16.109.176:8080 - R:/172.16.109.176:53656] Decreasing pending responses, now 0
2020-11-10 04:01:27.564 DEBUG 1 --- [or-http-epoll-3] r.n.http.server.HttpServerOperations : [id: 0x0473af0b, L:/172.16.109.176:8080 - R:/172.16.109.176:53656] Last HTTP packet was sent, terminating the channel
2020-11-10 07:10:47.140 DEBUG 1 --- [or-http-epoll-4] r.n.http.server.HttpServerOperations : [id: 0x0f2a88f3, L:/172.16.109.176:8080 - R:/10.213.43.170:13217] Increasing pending responses, now 1
2020-11-10 07:10:47.141 DEBUG 1 --- [or-http-epoll-4] reactor.netty.http.server.HttpServer : [id: 0x0f2a88f3, L:/172.16.109.176:8080 - R:/10.213.43.170:13217] Handler is being applied: org.springframework.http.server.reactive.ReactorHttpHandlerAdapter@209184fc
2020-11-10 07:10:47.142 DEBUG 1 --- [or-http-epoll-4] o.s.w.s.adapter.HttpWebHandlerAdapter : [0f2a88f3] HTTP GET "/v2.0/ports?device_id=c9cf7c56-81b5-4053-a0da-78aae0053806&device_id=49213ae4-59f0-4d75-bef3-2c8b1397e536&device_id=2be21821-2ccf-40a2-b33c-b0562f982cc7&device_id=303c1d5b-f432-471c-a765-c38ba5b6374d&device_id=f92407f6-665a-4cab-a0bb-b656cfa92367&device_id=b3f1e7b0-7d05-4da1-8f0b-15f5e4f4dae7"
2020-11-10 07:10:47.144 DEBUG 1 --- [or-http-epoll-4] o.s.c.g.h.RoutePredicateHandlerMapping : Route matched: beb6f9d5-1b8c-4d54-bc6b-520ccffeb1ab
2020-11-10 07:10:47.144 DEBUG 1 --- [or-http-epoll-4] o.s.c.g.h.RoutePredicateHandlerMapping : Mapping [Exchange: GET http://10.213.43.170:30009/v2.0/ports?device_id=c9cf7c56-81b5-4053-a0da-78aae0053806&device_id=49213ae4-59f0-4d75-bef3-2c8b1397e536&device_id=2be21821-2ccf-40a2-b33c-b0562f982cc7&device_id=303c1d5b-f432-471c-a765-c38ba5b6374d&device_id=f92407f6-665a-4cab-a0bb-b656cfa92367&device_id=b3f1e7b0-7d05-4da1-8f0b-15f5e4f4dae7] to Route{id='beb6f9d5-1b8c-4d54-bc6b-520ccffeb1ab', uri=http://portmanager-service.default.svc.cluster.local:9006/, order=0, predicate=org.springframework.cloud.gateway.support.ServerWebExchangeUtils$$Lambda$389/0x00000008404b0c40@547a5d53, gatewayFilters=[]}
2020-11-10 07:10:47.144 DEBUG 1 --- [or-http-epoll-4] o.s.c.g.h.RoutePredicateHandlerMapping : [0f2a88f3] Mapped to org.springframework.cloud.gateway.handler.FilteringWebHandler@6d5eb6c7
2020-11-10 07:10:47.144 DEBUG 1 --- [or-http-epoll-4] o.s.c.g.handler.FilteringWebHandler : Sorted gatewayFilterFactories: [OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.RemoveCachedBodyFilter@573284a5}, order=-2147483648}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.AdaptCachedBodyGlobalFilter@5e268ce6}, order=-2147482648}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=com.futurewei.alcor.apigateway.filter.KeystoneAuthGwFilter@6601cc93}, order=-100}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.NettyWriteResponseFilter@63716833}, order=-1}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.ForwardPathFilter@4bafe935}, order=0}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.GatewayMetricsFilter@4a7a965d}, order=0}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.RouteToRequestUrlFilter@3fbfbf84}, order=10000}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.WebsocketRoutingFilter@87b5b49}, order=2147483646}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.NettyRoutingFilter@54d901aa}, order=2147483647}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.ForwardRoutingFilter@23f72d88}, order=2147483647}]
2020-11-10 07:10:47.145 DEBUG 1 --- [or-http-epoll-4] c.f.a.a.filter.KeystoneAuthGwFilter : incoming request headers: [Host:"10.213.43.170:30009", User-Agent:"python-neutronclient", Accept-Encoding:"gzip, deflate", Accept:"application/json", Connection:"keep-alive", X-OpenStack-Request-ID:"req-d8733e4b-f357-44de-88a2-250dd38df5dc", X-Auth-Token:"gAAAAABfqjzoQQdhVR0o1S-CF34qqaqpcI1oMLzEfmUSZof2nGv_q-leTT6JVoxzqEGDGayPkUm2H6E2EYtC4MLDHR3n2jKBfrw8c6jWucZP-GKA9YDVLd06DlOtHJLo9lygGJEOTesfibfe7CIWD23EUCyR19VcjWOiZAvrbbqpRGPOS_mq41NnPQbriJv6mFcbb9QXCUO-", X-Service-Token:"gAAAAABfqdkM49YvF3UxSZifw2uU71eC82gcB-0yhGRK2JezX1cYLRt_yPGq0RU0e4CnmFjTl7KqNOw0Y_Chl7nrbeky6oD7XMq-dXHRNFWLjYDVPdA29tRFAgQ5fl879zaX2bPPA9piHrNAuVfwuKLKgGEtDn0hSml4Tb_ZFC9H_kNtCjKxOko"]@xieus I found the problem. In spring gateway the url line is limit to 4096 and http header limit to 8192.
Gzure
commented
3 years ago If we need to send this large URL, we should add a custom configuration class to augment the limit.
xieus
commented
3 years ago True but this limit has been lifted by PR #465. Besides, if we start a API GW locally, the same payload is working all the time which suggests that this may not be an functionality issue (otherwise, it will be 100% reproducible).
xieus
commented
3 years ago This has more comprehensive logs with error at the end.
2020-11-10 03:09:02.082 DEBUG 1 --- [or-http-epoll-3] r.n.http.server.HttpServerOperations : [id: 0xfa45935b, L:/172.16.109.176:8080 - R:/172.16.206.192:29830] Last HTTP response frame
2020-11-10 03:09:02.082 DEBUG 1 --- [or-http-epoll-3] r.n.http.server.HttpServerOperations : [id: 0xfa45935b, L:/172.16.109.176:8080 - R:/172.16.206.192:29830] No sendHeaders() called before complete, sending zero-length header
2020-11-10 03:09:02.083 DEBUG 1 --- [or-http-epoll-3] r.n.http.server.HttpServerOperations : [id: 0xfa45935b, L:/172.16.109.176:8080 - R:/172.16.206.192:29830] Decreasing pending responses, now 0
2020-11-10 03:09:02.083 DEBUG 1 --- [or-http-epoll-3] r.n.http.server.HttpServerOperations : [id: 0xfa45935b, L:/172.16.109.176:8080 - R:/172.16.206.192:29830] Last HTTP packet was sent, terminating the channel
2020-11-10 03:09:02.083 DEBUG 1 --- [or-http-epoll-3] r.n.channel.ChannelOperationsHandler : [id: 0xfa45935b, L:/172.16.109.176:8080 - R:/172.16.206.192:29830] No ChannelOperation attached. Dropping: EmptyLastHttpContent
2020-11-10 04:01:27.549 DEBUG 1 --- [or-http-epoll-3] r.n.http.server.HttpServerOperations : [id: 0x0473af0b, L:/172.16.109.176:8080 - R:/172.16.109.176:53656] New http connection, requesting read
2020-11-10 04:01:27.549 DEBUG 1 --- [or-http-epoll-3] reactor.netty.channel.BootstrapHandlers : [id: 0x0473af0b, L:/172.16.109.176:8080 - R:/172.16.109.176:53656] Initialized pipeline DefaultChannelPipeline{(BootstrapHandlers$BootstrapInitializerHandler#0 = reactor.netty.channel.BootstrapHandlers$BootstrapInitializerHandler), (reactor.left.httpCodec = io.netty.handler.codec.http.HttpServerCodec), (reactor.left.httpTrafficHandler = reactor.netty.http.server.HttpTrafficHandler), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
2020-11-10 04:01:27.550 DEBUG 1 --- [or-http-epoll-3] r.n.http.server.HttpServerOperations : [id: 0x0473af0b, L:/172.16.109.176:8080 - R:/172.16.109.176:53656] Increasing pending responses, now 1
2020-11-10 04:01:27.550 DEBUG 1 --- [or-http-epoll-3] reactor.netty.http.server.HttpServer : [id: 0x0473af0b, L:/172.16.109.176:8080 - R:/172.16.109.176:53656] Handler is being applied: org.springframework.http.server.reactive.ReactorHttpHandlerAdapter@209184fc
2020-11-10 04:01:27.550 DEBUG 1 --- [or-http-epoll-3] o.s.w.s.adapter.HttpWebHandlerAdapter : [0473af0b] HTTP GET "/project/f6f8eb223c114f0cb3aa965328c7f1a6/ports?device_id=2d2caba5-ee35-4c74-850f-19793481bdfb&device_id=a0168461-f67c-4a6a-b3c8-1045f1982f83&device_id=c674fa4f-6428-4ccf-968c-50f98c6e334e&device_id=620c173f-03ce-4285-ab67-c5ff214a37d9&device_id=f1247529-a762-4139-abbc-8246278bed21&device_id=0d25ad74-ce8c-4c33-8f68-9b3558fcec11&device_id=f47aa0dd-3c68-4367-adb9-9c86474fb746&device_id=91884ae0-f6ce-4b0c-990c-8426502d961e&device_id=fa4538fa-1010-4c64-989e-a6afe2c4f425&device_id=a3418f5d-bcfb-4b8a-a6b9-acb886cff858&device_id=542207cd-27f3-4523-88bb-6dfc2f2aaa2e&device_id=195975ee-c83c-4e43-b770-0b2dc0563956&device_id=c47c37e6-a96e-4028-9af1-6c48bcb486cc&device_id=45e1b11a-0cf8-4e0c-94d6-6cadce49d548&device_id=828e0aad-ec1c-4ce8-b979-51669dcf7fb7&device_id=cabcc7c2-5f10-4418-b4dd-5dd6c50b02c5&device_id=6db2adf1-f600-4f5e-a0be-17619e1bdae7&device_id=b4a0ad16-92f4-4a17-acfe-4a4ba915293d&device_id=fc6b79d8-fc84-40e8-89ac-72eee64a0d65&device_id=c2cbccf9-b5aa-4b3f-9ac0-a5d9394c93ec&device_id=95fd1ca3-cafe-423f-8a70-65a01fb4c033&device_id=5bbc2ed3-3513-4dba-99ed-1a6f4a47df2b&device_id=edf856d1-df23-4d74-8350-508c071b35b5&device_id=5f4bf89b-3d83-4426-bcfc-9829490a22cf&device_id=16e62417-e336-476a-8745-52266a4b278c&device_id=0863e0cd-2e8b-4e9d-8479-f8a3d8ce53a6&device_id=8532d38f-9e57-4e15-a0db-80c54f7820ac&device_id=f8b538b9-8744-4045-8d7c-a666a163a81f&device_id=2807af46-e128-4419-abe9-0ed7fe6b4556&device_id=77e28984-7055-4ce2-a8f5-cc064ec27d47&device_id=93ebc33a-9a2c-41ec-b1f2-b56d31a0ed26&device_id=a39d4130-99f5-4130-b5e1-5ba98ce12c2a&device_id=108858d2-f9b5-47a5-8999-2e36b4903b16&device_id=bd997f95-28fd-4817-a7a0-da09a34af305&device_id=fce7479b-79ef-49cc-a902-4d29ad0fec89&device_id=da9e8e0b-d379-4551-ada9-2cc4a3c05c42&device_id=74ee0a00-155a-49cd-8000-098e1b0630cb&device_id=4f03669b-0915-442c-a3ec-d7755400203b&device_id=e28e6892-1710-495a-87df-972b39267c27&device_id=5af14455-8c8f-4f63-a2ff-e39a2d60466d&device_id=7a374323-ff11-46cc-8425-cbe881b448ef&device_id=8fee06fa-3f30-415f-9a92-cf14cb452625&device_id=92d6c2e6-fc04-47c2-aea4-41ac2924a9ec&device_id=d7b8b420-c49a-4715-843e-b515b1fef975&device_id=92b3626c-845b-4a1c-9eef-852382b9cd29&device_id=27848d59-389e-4f05-8abb-ef94d42e1128&device_id=2fb2d76b-e6d2-4ffe-94ac-e161709ad9b5&device_id=ed46a188-76d8-4149-bdcb-d1271c659111&device_id=ef7ccb72-a15f-4683-b3dc-060fcb927842&device_id=633ccc71-b33a-489d-ac52-413719b7c91e&device_id=560d847a-b0cb-40bf-aa8d-4119f88190ca&device_id=c6d089df-a662-4083-af1c-2db3de3d74de&device_id=a3c73407-811b-4c4d-966d-e12e870c1582&device_id=745bdb22-df1f-4d30-a8cc-6c5d6e48b4fb&device_id=133e62cd-0ccc-4581-b767-88c5729e31fb&device_id=a0bd2545-9e7e-46c2-a450-0a96fbbc532e&device_id=abc46ccd-82ae-4202-9f1e-fdc8b2bcbb19&device_id=9e84dd65-56f9-4a55-a931-166a095de62a&device_id=be888734-e953-4f04-b801-1597cdacaad2&device_id=23ad6e3f-60b8-46b1-95e2-d2634072b09a&device_id=b021d095-9144-406f-90f6-77da4c449671&device_id=66163c5b-e338-48c9-ad33-86b43fa51ab9&device_id=5e546575-dc12-4013-b630-961b64d5a369&device_id=e772a9d2-f731-4ce6-a68e-79523ac72dfb&device_id=171ece2b-8b43-4cde-b966-5133e9e6ce2b&device_id=b3d377d9-9440-4e13-b941-fb76c3024ee3&device_id=53706fd0-06bc-4f76-a5fc-e925cea99df7&device_id=2720fd9a-ed9b-4788-9d82-27bff7df9cdc&device_id=406ff289-d63d-4f70-8bb6-af3631308980&device_id=47f6d477-37ff-40d2-9e13-c2a2227f4f70&device_id=d9b4cd8e-1023-4f64-9fe1-25e1f8f23d32&device_id=cbe4e7d3-e4d4-4225-bfbe-192d72ad854d&device_id=157f9316-9776-4b7f-ac8d-ad7b92bd8831&device_id=da9802de-3dbc-40ec-97a1-b973d34b66f1&device_id=6c1688fe-19bb-49b1-ae61-1b63687f7887&device_id=8829c3ca-e49b-474f-b034-0380d124c292&device_id=ef43665f-4b9e-464c-85a8-f16cfb93d082&device_id=ec969faa-eb5d-45ce-b736-c29de21784c3&device_id=340f1a6b-3625-45f5-8fba-5e22e1cc3f78&device_id=40fc9d52-92f4-43ca-87bd-3c0a93a3fcbf&device_id=90776e20-eff6-42b2-9560-a57f4727f724&device_id=681d0727-15c5-4fe7-a809-c78012a090c4&device_id=ad4f1fba-0003-438a-8fab-55ee9dd59a83&device_id=3d316c2c-50a9-4883-b850-a083e2f30f5b&device_id=e6d53118-edab-465c-9d89-17aa44f774bf&device_id=ee0e47e4-66dc-4cee-ae1d-5df483e32f45&device_id=748cbe7d-e66c-4d26-ac0f-ff95cbb6c9dd&device_id=24badaf2-07e4-4400-b595-bb7a02ec0b72&device_id=b544a79f-0852-4fe5-8a8a-53254fb5e9f1&device_id=15381f9d-7e9e-4d0c-b995-4423ab27ebde&device_id=315a22f9-ea01-4697-97a0-294493542f31&device_id=8782704f-54c0-44c3-b04a-ca6c42f22dc0&device_id=2ab2c5e7-08a7-44b5-a1fe-cd6d0f12bb58&device_id=bc8db389-29c0-40f7-aafd-96f2bdd77859&device_id=2f58c899-8527-448e-91e5-d689b7e464a9&device_id=eaea5d2b-275b-4513-9493-36605b34fb1b&device_id=e420641f-7c07-498c-a6d6-47fec7f1a933&device_id=733ab157-4841-4a27-9dcb-751c29ac97be&device_id=41c6908b-c25b-4ec2-92c2-149e910b8a41&device_id=b8187e1b-f862-4be3-89ed-49ebf667abc7&device_id=36c49870-58c6-42ec-874b-bd0ab9302e86&device_id=03dd5d38-52a9-45af-aaae-0bc235e6f1ff&device_id=b64459b1-2163-4b76-82ee-16bf7d35619f&device_id=97fe02a1-8116-4f97-8926-193bad57be05&device_id=143d80ac-0d4a-498e-8e9a-80d865483c6f&device_id=7190d587-af64-4d53-a0e3-c201edc384ef&device_id=f519fb0a-67d1-4eb4-bb62-0fe34c8e71c4&device_id=5f390f3e-6992-45a5-967d-68ce73acbc3b&device_id=71e28f9e-04f2-4431-a183-e619eddbe559&device_id=a11b5619-bdd5-487c-9306-dd037529cf12&device_id=cbf2de40-9ec4-459d-82a3-9feafee72b47&device_id=a880a472-2705-4d31-948c-0a9d0fd1e067&device_id=2c2b1e76-2bb5-4e5e-a8c5-37fa2ec710df&device_id=2f58387a-4295-48f2-935f-6b097af50740&device_id=feb458c0-d9b4-4c41-9431-aa6dd3763780&device_id=368e8a70-b586-4e67-83bf-9b51a7fc39dd&device_id=d8a2ebd9-1967-4cba-8394-8ff96da0327a&device_id=4594aab7-61c2-4233-b27a-3e390e0118b2&device_id=d65932bf-2123-4d41-a58f-0dbe29d5415e&device_id=e8f20d40-43e2-49c4-8ad3-3498e9d2bff0&device_id=02f4b757-2536-4ea7-8700-74594cb5de57&device_id=63012149-ae4f-4713-b4f3-d48f53efeebd&device_id=480f1180-8e7a-48e4-85d2-4194dc82ec55&device_id=8b6fb21a-ecca-4a32-a93c-144c0c86196a&device_id=eb6c06ad-6b12-4fa8-87c4-0ea5fe43ab6f&device_id=db311029-9dc4-46bd-bc3b-c7dc6e35725c&device_id=b3cae8ce-cc30-4ce5-b17d-6a6184aebaf4&device_id=28b62b76-af05-4115-9c06-3feb14c18e33&device_id=f90b94f7-72b2-47e1-883a-ed465c7aa9b3&device_id=874eb6b8-fce6-4c19-a402-026eb93c0399&device_id=2b99b7a0-5a31-4a97-a786-84eede30e638&device_id=04658867-e7d4-4564-9b8d-25b520fabda7&device_id=ed9832ef-9d59-416f-96eb-4d9774d4ae20&device_id=3f72b077-069a-4c5b-8f29-0c48b5c15e0f&device_id=dd0104c4-d6ae-41ac-b403-2a08742abbca&device_id=9958306f-2dd1-4835-a4d1-dc1cf19a890c&device_id=d0d6e514-796c-4a97-a9b5-e915351ae83d&device_id=afa9834a-fea3-45f1-a5b1-7b1e4464f477&device_id=997d7701-f769-4df9-8029-164b808a7ed1&device_id=b617e04b-b08b-4408-9541-9047d410d10c&device_id=ca2d692d-759c-4184-a7bd-ef83465f64e6&device_id=79824d01-2cb1-4b05-b881-2cf674ca2df4&device_id=2a1f8813-45e3-48ad-a4cb-c372f3153cde&device_id=6833cda4-3684-44a3-bfac-cf1a2bf9f99d&device_id=00529586-6b29-468b-845f-99d8e340552c&device_id=da93586d-521c-46b9-bc54-46ca5692b349&device_id=196196e2-2f1f-41fb-b5fc-da5171967c1f&device_id=d655f018-6573-4eff-ba2d-312143e71370&device_id=df0b6b2d-4b73-4486-b897-9ce29d15fe43&device_id=f63fb2ad-0158-40b2-9a9a-f68e984c2fff"
2020-11-10 04:01:27.551 DEBUG 1 --- [or-http-epoll-3] o.s.c.g.h.RoutePredicateHandlerMapping : Route matched: beb6f9d5-1b8c-4d54-bc6b-520ccffeb1ab
2020-11-10 04:01:27.551 DEBUG 1 --- [or-http-epoll-3] o.s.c.g.h.RoutePredicateHandlerMapping : Mapping [Exchange: GET http://172.16.109.176:8080/project/f6f8eb223c114f0cb3aa965328c7f1a6/ports?device_id=2d2caba5-ee35-4c74-850f-19793481bdfb&device_id=a0168461-f67c-4a6a-b3c8-1045f1982f83&device_id=c674fa4f-6428-4ccf-968c-50f98c6e334e&device_id=620c173f-03ce-4285-ab67-c5ff214a37d9&device_id=f1247529-a762-4139-abbc-8246278bed21&device_id=0d25ad74-ce8c-4c33-8f68-9b3558fcec11&device_id=f47aa0dd-3c68-4367-adb9-9c86474fb746&device_id=91884ae0-f6ce-4b0c-990c-8426502d961e&device_id=fa4538fa-1010-4c64-989e-a6afe2c4f425&device_id=a3418f5d-bcfb-4b8a-a6b9-acb886cff858&device_id=542207cd-27f3-4523-88bb-6dfc2f2aaa2e&device_id=195975ee-c83c-4e43-b770-0b2dc0563956&device_id=c47c37e6-a96e-4028-9af1-6c48bcb486cc&device_id=45e1b11a-0cf8-4e0c-94d6-6cadce49d548&device_id=828e0aad-ec1c-4ce8-b979-51669dcf7fb7&device_id=cabcc7c2-5f10-4418-b4dd-5dd6c50b02c5&device_id=6db2adf1-f600-4f5e-a0be-17619e1bdae7&device_id=b4a0ad16-92f4-4a17-acfe-4a4ba915293d&device_id=fc6b79d8-fc84-40e8-89ac-72eee64a0d65&device_id=c2cbccf9-b5aa-4b3f-9ac0-a5d9394c93ec&device_id=95fd1ca3-cafe-423f-8a70-65a01fb4c033&device_id=5bbc2ed3-3513-4dba-99ed-1a6f4a47df2b&device_id=edf856d1-df23-4d74-8350-508c071b35b5&device_id=5f4bf89b-3d83-4426-bcfc-9829490a22cf&device_id=16e62417-e336-476a-8745-52266a4b278c&device_id=0863e0cd-2e8b-4e9d-8479-f8a3d8ce53a6&device_id=8532d38f-9e57-4e15-a0db-80c54f7820ac&device_id=f8b538b9-8744-4045-8d7c-a666a163a81f&device_id=2807af46-e128-4419-abe9-0ed7fe6b4556&device_id=77e28984-7055-4ce2-a8f5-cc064ec27d47&device_id=93ebc33a-9a2c-41ec-b1f2-b56d31a0ed26&device_id=a39d4130-99f5-4130-b5e1-5ba98ce12c2a&device_id=108858d2-f9b5-47a5-8999-2e36b4903b16&device_id=bd997f95-28fd-4817-a7a0-da09a34af305&device_id=fce7479b-79ef-49cc-a902-4d29ad0fec89&device_id=da9e8e0b-d379-4551-ada9-2cc4a3c05c42&device_id=74ee0a00-155a-49cd-8000-098e1b0630cb&device_id=4f03669b-0915-442c-a3ec-d7755400203b&device_id=e28e6892-1710-495a-87df-972b39267c27&device_id=5af14455-8c8f-4f63-a2ff-e39a2d60466d&device_id=7a374323-ff11-46cc-8425-cbe881b448ef&device_id=8fee06fa-3f30-415f-9a92-cf14cb452625&device_id=92d6c2e6-fc04-47c2-aea4-41ac2924a9ec&device_id=d7b8b420-c49a-4715-843e-b515b1fef975&device_id=92b3626c-845b-4a1c-9eef-852382b9cd29&device_id=27848d59-389e-4f05-8abb-ef94d42e1128&device_id=2fb2d76b-e6d2-4ffe-94ac-e161709ad9b5&device_id=ed46a188-76d8-4149-bdcb-d1271c659111&device_id=ef7ccb72-a15f-4683-b3dc-060fcb927842&device_id=633ccc71-b33a-489d-ac52-413719b7c91e&device_id=560d847a-b0cb-40bf-aa8d-4119f88190ca&device_id=c6d089df-a662-4083-af1c-2db3de3d74de&device_id=a3c73407-811b-4c4d-966d-e12e870c1582&device_id=745bdb22-df1f-4d30-a8cc-6c5d6e48b4fb&device_id=133e62cd-0ccc-4581-b767-88c5729e31fb&device_id=a0bd2545-9e7e-46c2-a450-0a96fbbc532e&device_id=abc46ccd-82ae-4202-9f1e-fdc8b2bcbb19&device_id=9e84dd65-56f9-4a55-a931-166a095de62a&device_id=be888734-e953-4f04-b801-1597cdacaad2&device_id=23ad6e3f-60b8-46b1-95e2-d2634072b09a&device_id=b021d095-9144-406f-90f6-77da4c449671&device_id=66163c5b-e338-48c9-ad33-86b43fa51ab9&device_id=5e546575-dc12-4013-b630-961b64d5a369&device_id=e772a9d2-f731-4ce6-a68e-79523ac72dfb&device_id=171ece2b-8b43-4cde-b966-5133e9e6ce2b&device_id=b3d377d9-9440-4e13-b941-fb76c3024ee3&device_id=53706fd0-06bc-4f76-a5fc-e925cea99df7&device_id=2720fd9a-ed9b-4788-9d82-27bff7df9cdc&device_id=406ff289-d63d-4f70-8bb6-af3631308980&device_id=47f6d477-37ff-40d2-9e13-c2a2227f4f70&device_id=d9b4cd8e-1023-4f64-9fe1-25e1f8f23d32&device_id=cbe4e7d3-e4d4-4225-bfbe-192d72ad854d&device_id=157f9316-9776-4b7f-ac8d-ad7b92bd8831&device_id=da9802de-3dbc-40ec-97a1-b973d34b66f1&device_id=6c1688fe-19bb-49b1-ae61-1b63687f7887&device_id=8829c3ca-e49b-474f-b034-0380d124c292&device_id=ef43665f-4b9e-464c-85a8-f16cfb93d082&device_id=ec969faa-eb5d-45ce-b736-c29de21784c3&device_id=340f1a6b-3625-45f5-8fba-5e22e1cc3f78&device_id=40fc9d52-92f4-43ca-87bd-3c0a93a3fcbf&device_id=90776e20-eff6-42b2-9560-a57f4727f724&device_id=681d0727-15c5-4fe7-a809-c78012a090c4&device_id=ad4f1fba-0003-438a-8fab-55ee9dd59a83&device_id=3d316c2c-50a9-4883-b850-a083e2f30f5b&device_id=e6d53118-edab-465c-9d89-17aa44f774bf&device_id=ee0e47e4-66dc-4cee-ae1d-5df483e32f45&device_id=748cbe7d-e66c-4d26-ac0f-ff95cbb6c9dd&device_id=24badaf2-07e4-4400-b595-bb7a02ec0b72&device_id=b544a79f-0852-4fe5-8a8a-53254fb5e9f1&device_id=15381f9d-7e9e-4d0c-b995-4423ab27ebde&device_id=315a22f9-ea01-4697-97a0-294493542f31&device_id=8782704f-54c0-44c3-b04a-ca6c42f22dc0&device_id=2ab2c5e7-08a7-44b5-a1fe-cd6d0f12bb58&device_id=bc8db389-29c0-40f7-aafd-96f2bdd77859&device_id=2f58c899-8527-448e-91e5-d689b7e464a9&device_id=eaea5d2b-275b-4513-9493-36605b34fb1b&device_id=e420641f-7c07-498c-a6d6-47fec7f1a933&device_id=733ab157-4841-4a27-9dcb-751c29ac97be&device_id=41c6908b-c25b-4ec2-92c2-149e910b8a41&device_id=b8187e1b-f862-4be3-89ed-49ebf667abc7&device_id=36c49870-58c6-42ec-874b-bd0ab9302e86&device_id=03dd5d38-52a9-45af-aaae-0bc235e6f1ff&device_id=b64459b1-2163-4b76-82ee-16bf7d35619f&device_id=97fe02a1-8116-4f97-8926-193bad57be05&device_id=143d80ac-0d4a-498e-8e9a-80d865483c6f&device_id=7190d587-af64-4d53-a0e3-c201edc384ef&device_id=f519fb0a-67d1-4eb4-bb62-0fe34c8e71c4&device_id=5f390f3e-6992-45a5-967d-68ce73acbc3b&device_id=71e28f9e-04f2-4431-a183-e619eddbe559&device_id=a11b5619-bdd5-487c-9306-dd037529cf12&device_id=cbf2de40-9ec4-459d-82a3-9feafee72b47&device_id=a880a472-2705-4d31-948c-0a9d0fd1e067&device_id=2c2b1e76-2bb5-4e5e-a8c5-37fa2ec710df&device_id=2f58387a-4295-48f2-935f-6b097af50740&device_id=feb458c0-d9b4-4c41-9431-aa6dd3763780&device_id=368e8a70-b586-4e67-83bf-9b51a7fc39dd&device_id=d8a2ebd9-1967-4cba-8394-8ff96da0327a&device_id=4594aab7-61c2-4233-b27a-3e390e0118b2&device_id=d65932bf-2123-4d41-a58f-0dbe29d5415e&device_id=e8f20d40-43e2-49c4-8ad3-3498e9d2bff0&device_id=02f4b757-2536-4ea7-8700-74594cb5de57&device_id=63012149-ae4f-4713-b4f3-d48f53efeebd&device_id=480f1180-8e7a-48e4-85d2-4194dc82ec55&device_id=8b6fb21a-ecca-4a32-a93c-144c0c86196a&device_id=eb6c06ad-6b12-4fa8-87c4-0ea5fe43ab6f&device_id=db311029-9dc4-46bd-bc3b-c7dc6e35725c&device_id=b3cae8ce-cc30-4ce5-b17d-6a6184aebaf4&device_id=28b62b76-af05-4115-9c06-3feb14c18e33&device_id=f90b94f7-72b2-47e1-883a-ed465c7aa9b3&device_id=874eb6b8-fce6-4c19-a402-026eb93c0399&device_id=2b99b7a0-5a31-4a97-a786-84eede30e638&device_id=04658867-e7d4-4564-9b8d-25b520fabda7&device_id=ed9832ef-9d59-416f-96eb-4d9774d4ae20&device_id=3f72b077-069a-4c5b-8f29-0c48b5c15e0f&device_id=dd0104c4-d6ae-41ac-b403-2a08742abbca&device_id=9958306f-2dd1-4835-a4d1-dc1cf19a890c&device_id=d0d6e514-796c-4a97-a9b5-e915351ae83d&device_id=afa9834a-fea3-45f1-a5b1-7b1e4464f477&device_id=997d7701-f769-4df9-8029-164b808a7ed1&device_id=b617e04b-b08b-4408-9541-9047d410d10c&device_id=ca2d692d-759c-4184-a7bd-ef83465f64e6&device_id=79824d01-2cb1-4b05-b881-2cf674ca2df4&device_id=2a1f8813-45e3-48ad-a4cb-c372f3153cde&device_id=6833cda4-3684-44a3-bfac-cf1a2bf9f99d&device_id=00529586-6b29-468b-845f-99d8e340552c&device_id=da93586d-521c-46b9-bc54-46ca5692b349&device_id=196196e2-2f1f-41fb-b5fc-da5171967c1f&device_id=d655f018-6573-4eff-ba2d-312143e71370&device_id=df0b6b2d-4b73-4486-b897-9ce29d15fe43&device_id=f63fb2ad-0158-40b2-9a9a-f68e984c2fff] to Route{id='beb6f9d5-1b8c-4d54-bc6b-520ccffeb1ab', uri=http://portmanager-service.default.svc.cluster.local:9006/, order=0, predicate=org.springframework.cloud.gateway.support.ServerWebExchangeUtils$$Lambda$389/0x00000008404b0c40@547a5d53, gatewayFilters=[]}
2020-11-10 04:01:27.551 DEBUG 1 --- [or-http-epoll-3] o.s.c.g.h.RoutePredicateHandlerMapping : [0473af0b] Mapped to org.springframework.cloud.gateway.handler.FilteringWebHandler@6d5eb6c7
2020-11-10 04:01:27.551 DEBUG 1 --- [or-http-epoll-3] o.s.c.g.handler.FilteringWebHandler : Sorted gatewayFilterFactories: [OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.RemoveCachedBodyFilter@573284a5}, order=-2147483648}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.AdaptCachedBodyGlobalFilter@5e268ce6}, order=-2147482648}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=com.futurewei.alcor.apigateway.filter.KeystoneAuthGwFilter@6601cc93}, order=-100}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.NettyWriteResponseFilter@63716833}, order=-1}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.ForwardPathFilter@4bafe935}, order=0}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.GatewayMetricsFilter@4a7a965d}, order=0}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.RouteToRequestUrlFilter@3fbfbf84}, order=10000}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.WebsocketRoutingFilter@87b5b49}, order=2147483646}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.NettyRoutingFilter@54d901aa}, order=2147483647}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.ForwardRoutingFilter@23f72d88}, order=2147483647}]
2020-11-10 04:01:27.551 DEBUG 1 --- [or-http-epoll-3] c.f.a.a.filter.KeystoneAuthGwFilter : incoming request headers: [User-Agent:"curl/7.29.0", Host:"172.16.109.176:8080", Accept:"*/*", cache-control:"no-cache", postman-token:"ecb994c7-b0cf-ed97-3a1a-ca666b6ee6e7", x-auth-token:"gAAAAABfqgPlDkNEUYkojOURplwc_9Tt6oaBlCRH6om3Ness6DFP3R9O5ND7PVhVhMBNnXuVItQF5yULqrO6OfB8YPAN_DGAy-eRL3k4xeTPunhjoqGpuBClbl8hIwpVIWo-g7oUbv9P2lWczri-ChCaz-nziJLsLFrNHE2Q4Lse2EPNqXOB-5A"]
2020-11-10 04:01:27.553 DEBUG 1 --- [or-http-epoll-3] c.f.a.apigateway.client.KeystoneClient : fetch the token from cache TokenEntity{token='gAAAAABfqgPlDkNEUYkojOURplwc_9Tt6oaBlCRH6om3Ness6DFP3R9O5ND7PVhVhMBNnXuVItQF5yULqrO6OfB8YPAN_DGAy-eRL3k4xeTPunhjoqGpuBClbl8hIwpVIWo-g7oUbv9P2lWczri-ChCaz-nziJLsLFrNHE2Q4Lse2EPNqXOB-5A', expireAt=Tue Nov 10 15:07:17 UTC 2020, user='admin1', userId='5179052cda854f1da2663c600f022e81', domainId='default', domainName='Default', projectName='admin1', projectId='5ddc6d5e405f4a8b8665bf2e812284f0', projectDomain='null', invalid=false, roles=[reader, admin, member]}
2020-11-10 04:01:27.553 DEBUG 1 --- [or-http-epoll-3] c.f.a.a.filter.KeystoneAuthGwFilter : parsed token gAAAAABfqgPlDkNEUYkojOURplwc_9Tt6oaBlCRH6om3Ness6DFP3R9O5ND7PVhVhMBNnXuVItQF5yULqrO6OfB8YPAN_DGAy-eRL3k4xeTPunhjoqGpuBClbl8hIwpVIWo-g7oUbv9P2lWczri-ChCaz-nziJLsLFrNHE2Q4Lse2EPNqXOB-5A project id success, project id:[5ddc6d5e405f4a8b8665bf2e812284f0]
2020-11-10 04:01:27.559 DEBUG 1 --- [or-http-epoll-3] reactor.netty.channel.FluxReceive : [id: 0x0473af0b, L:/172.16.109.176:8080 - R:/172.16.109.176:53656] Subscribing inbound receiver [pending: 0, cancelled:false, inboundDone: true]
2020-11-10 04:01:27.564 DEBUG 1 --- [or-http-epoll-3] o.s.w.s.adapter.HttpWebHandlerAdapter : [0473af0b] Completed 400 BAD_REQUEST
2020-11-10 04:01:27.564 DEBUG 1 --- [or-http-epoll-3] r.n.http.server.HttpServerOperations : [id: 0x0473af0b, L:/172.16.109.176:8080 - R:/172.16.109.176:53656] Last HTTP response frame
2020-11-10 04:01:27.564 DEBUG 1 --- [or-http-epoll-3] r.n.http.server.HttpServerOperations : [id: 0x0473af0b, L:/172.16.109.176:8080 - R:/172.16.109.176:53656] Decreasing pending responses, now 0
2020-11-10 04:01:27.564 DEBUG 1 --- [or-http-epoll-3] r.n.http.server.HttpServerOperations : [id: 0x0473af0b, L:/172.16.109.176:8080 - R:/172.16.109.176:53656] Last HTTP packet was sent, terminating the channelOk, let's set the log level to trace to obtain more debugging information, and more error 400 information can be obtained.
Gzure
commented
3 years ago @xieus Please set 'server.max-http-header-size=16KB' in PM, and test whether getting 400 error.
xieus
commented
3 years ago @Gzure Thanks. Yes you are right. After enabling the TRACE level, it showed PM returns 400 and further tracing back to PM found the error. Somehow the error was not found in last round of investigation (maybe by mistake). The fix has been verified by a manual API test.
xieus
commented
3 years ago Also, a quick fix is also included in PR #471. I will paste more logs here for future reference.
xieus
commented
3 years ago PM Logs
2020-11-11 05:30:27.539 DEBUG 1 --- [io-8080-exec-10] o.a.tomcat.util.net.SocketWrapperBase : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@745327ce:org.apache.tomcat.util.net.NioChannel@7046c733:java.nio.channels.SocketChannel[connected local=portmanager-748bcfbfb9-4f9r6/172.16.206.235:8080 remote=/172.16.109.168:46552]], Read from buffer: [0]
2020-11-11 05:30:27.539 DEBUG 1 --- [io-8080-exec-10] org.apache.tomcat.util.net.NioEndpoint : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@745327ce:org.apache.tomcat.util.net.NioChannel@7046c733:java.nio.channels.SocketChannel[connected local=portmanager-748bcfbfb9-4f9r6/172.16.206.235:8080 remote=/172.16.109.168:46552]], Read direct from socket: [8192]
2020-11-11 05:30:27.539 DEBUG 1 --- [io-8080-exec-10] o.a.coyote.http11.Http11InputBuffer : Received [GET /project/0c664c24649c4605aba77246d4b0c63f/ports?device_id=2d2caba5-ee35-4c74-850f-19793481bdfb&device_id=a0168461-f67c-4a6a-b3c8-1045f1982f83&device_id=c674fa4f-6428-4ccf-968c-50f98c6e334e&device_id=620c173f-03ce-4285-ab67-c5ff214a37d9&device_id=f1247529-a762-4139-abbc-8246278bed21&device_id=0d25ad74-ce8c-4c33-8f68-9b3558fcec11&device_id=f47aa0dd-3c68-4367-adb9-9c86474fb746&device_id=91884ae0-f6ce-4b0c-990c-8426502d961e&device_id=fa4538fa-1010-4c64-989e-a6afe2c4f425&device_id=a3418f5d-bcfb-4b8a-a6b9-acb886cff858&device_id=542207cd-27f3-4523-88bb-6dfc2f2aaa2e&device_id=195975ee-c83c-4e43-b770-0b2dc0563956&device_id=c47c37e6-a96e-4028-9af1-6c48bcb486cc&device_id=45e1b11a-0cf8-4e0c-94d6-6cadce49d548&device_id=828e0aad-ec1c-4ce8-b979-51669dcf7fb7&device_id=cabcc7c2-5f10-4418-b4dd-5dd6c50b02c5&device_id=6db2adf1-f600-4f5e-a0be-17619e1bdae7&device_id=b4a0ad16-92f4-4a17-acfe-4a4ba915293d&device_id=fc6b79d8-fc84-40e8-89ac-72eee64a0d65&device_id=c2cbccf9-b5aa-4b3f-9ac0-a5d9394c93ec&device_id=95fd1ca3-cafe-423f-8a70-65a01fb4c033&device_id=5bbc2ed3-3513-4dba-99ed-1a6f4a47df2b&device_id=edf856d1-df23-4d74-8350-508c071b35b5&device_id=5f4bf89b-3d83-4426-bcfc-9829490a22cf&device_id=16e62417-e336-476a-8745-52266a4b278c&device_id=0863e0cd-2e8b-4e9d-8479-f8a3d8ce53a6&device_id=8532d38f-9e57-4e15-a0db-80c54f7820ac&device_id=f8b538b9-8744-4045-8d7c-a666a163a81f&device_id=2807af46-e128-4419-abe9-0ed7fe6b4556&device_id=77e28984-7055-4ce2-a8f5-cc064ec27d47&device_id=93ebc33a-9a2c-41ec-b1f2-b56d31a0ed26&device_id=a39d4130-99f5-4130-b5e1-5ba98ce12c2a&device_id=108858d2-f9b5-47a5-8999-2e36b4903b16&device_id=bd997f95-28fd-4817-a7a0-da09a34af305&device_id=fce7479b-79ef-49cc-a902-4d29ad0fec89&device_id=da9e8e0b-d379-4551-ada9-2cc4a3c05c42&device_id=74ee0a00-155a-49cd-8000-098e1b0630cb&device_id=4f03669b-0915-442c-a3ec-d7755400203b&device_id=e28e6892-1710-495a-87df-972b39267c27&device_id=5af14455-8c8f-4f63-a2ff-e39a2d60466d&device_id=7a374323-ff11-46cc-8425-cbe881b448ef&device_id=8fee06fa-3f30-415f-9a92-cf14cb452625&device_id=92d6c2e6-fc04-47c2-aea4-41ac2924a9ec&device_id=d7b8b420-c49a-4715-843e-b515b1fef975&device_id=92b3626c-845b-4a1c-9eef-852382b9cd29&device_id=27848d59-389e-4f05-8abb-ef94d42e1128&device_id=2fb2d76b-e6d2-4ffe-94ac-e161709ad9b5&device_id=ed46a188-76d8-4149-bdcb-d1271c659111&device_id=ef7ccb72-a15f-4683-b3dc-060fcb927842&device_id=633ccc71-b33a-489d-ac52-413719b7c91e&device_id=560d847a-b0cb-40bf-aa8d-4119f88190ca&device_id=c6d089df-a662-4083-af1c-2db3de3d74de&device_id=a3c73407-811b-4c4d-966d-e12e870c1582&device_id=745bdb22-df1f-4d30-a8cc-6c5d6e48b4fb&device_id=133e62cd-0ccc-4581-b767-88c5729e31fb&device_id=a0bd2545-9e7e-46c2-a450-0a96fbbc532e&device_id=abc46ccd-82ae-4202-9f1e-fdc8b2bcbb19&device_id=9e84dd65-56f9-4a55-a931-166a095de62a&device_id=be888734-e953-4f04-b801-1597cdacaad2&device_id=23ad6e3f-60b8-46b1-95e2-d2634072b09a&device_id=b021d095-9144-406f-90f6-77da4c449671&device_id=66163c5b-e338-48c9-ad33-86b43fa51ab9&device_id=5e546575-dc12-4013-b630-961b64d5a369&device_id=e772a9d2-f731-4ce6-a68e-79523ac72dfb&device_id=171ece2b-8b43-4cde-b966-5133e9e6ce2b&device_id=b3d377d9-9440-4e13-b941-fb76c3024ee3&device_id=53706fd0-06bc-4f76-a5fc-e925cea99df7&device_id=2720fd9a-ed9b-4788-9d82-27bff7df9cdc&device_id=406ff289-d63d-4f70-8bb6-af3631308980&device_id=47f6d477-37ff-40d2-9e13-c2a2227f4f70&device_id=d9b4cd8e-1023-4f64-9fe1-25e1f8f23d32&device_id=cbe4e7d3-e4d4-4225-bfbe-192d72ad854d&device_id=157f9316-9776-4b7f-ac8d-ad7b92bd8831&device_id=da9802de-3dbc-40ec-97a1-b973d34b66f1&device_id=6c1688fe-19bb-49b1-ae61-1b63687f7887&device_id=8829c3ca-e49b-474f-b034-0380d124c292&device_id=ef43665f-4b9e-464c-85a8-f16cfb93d082&device_id=ec969faa-eb5d-45ce-b736-c29de21784c3&device_id=340f1a6b-3625-45f5-8fba-5e22e1cc3f78&device_id=40fc9d52-92f4-43ca-87bd-3c0a93a3fcbf&device_id=90776e20-eff6-42b2-9560-a57f4727f724&device_id=681d0727-15c5-4fe7-a809-c78012a090c4&device_id=ad4f1fba-0003-438a-8fab-55ee9dd59a83&device_id=3d316c2c-50a9-4883-b850-a083e2f30f5b&device_id=e6d53118-edab-465c-9d89-17aa44f774bf&device_id=ee0e47e4-66dc-4cee-ae1d-5df483e32f45&device_id=748cbe7d-e66c-4d26-ac0f-ff95cbb6c9dd&device_id=24badaf2-07e4-4400-b595-bb7a02ec0b72&device_id=b544a79f-0852-4fe5-8a8a-53254fb5e9f1&device_id=15381f9d-7e9e-4d0c-b995-4423ab27ebde&device_id=315a22f9-ea01-4697-97a0-294493542f31&device_id=8782704f-54c0-44c3-b04a-ca6c42f22dc0&device_id=2ab2c5e7-08a7-44b5-a1fe-cd6d0f12bb58&device_id=bc8db389-29c0-40f7-aafd-96f2bdd77859&device_id=2f58c899-8527-448e-91e5-d689b7e464a9&device_id=eaea5d2b-275b-4513-9493-36605b34fb1b&device_id=e420641f-7c07-498c-a6d6-47fec7f1a933&device_id=733ab157-4841-4a27-9dcb-751c29ac97be&device_id=41c6908b-c25b-4ec2-92c2-149e910b8a41&device_id=b8187e1b-f862-4be3-89ed-49ebf667abc7&device_id=36c49870-58c6-42ec-874b-bd0ab9302e86&device_id=03dd5d38-52a9-45af-aaae-0bc235e6f1ff&device_id=b64459b1-2163-4b76-82ee-16bf7d35619f&device_id=97fe02a1-8116-4f97-8926-193bad57be05&device_id=143d80ac-0d4a-498e-8e9a-80d865483c6f&device_id=7190d587-af64-4d53-a0e3-c201edc384ef&device_id=f519fb0a-67d1-4eb4-bb62-0fe34c8e71c4&device_id=5f390f3e-6992-45a5-967d-68ce73acbc3b&device_id=71e28f9e-04f2-4431-a183-e619eddbe559&device_id=a11b5619-bdd5-487c-9306-dd037529cf12&device_id=cbf2de40-9ec4-459d-82a3-9feafee72b47&device_id=a880a472-2705-4d31-948c-0a9d0fd1e067&device_id=2c2b1e76-2bb5-4e5e-a8c5-37fa2ec710df&device_id=2f58387a-4295-48f2-935f-6b097af50740&device_id=feb458c0-d9b4-4c41-9431-aa6dd3763780&device_id=368e8a70-b586-4e67-83bf-9b51a7fc39dd&device_id=d8a2ebd9-1967-4cba-8394-8ff96da0327a&device_id=4594aab7-61c2-4233-b27a-3e390e0118b2&device_id=d65932bf-2123-4d41-a58f-0dbe29d5415e&device_id=e8f20d40-43e2-49c4-8ad3-3498e9d2bff0&device_id=02f4b757-2536-4ea7-8700-74594cb5de57&device_id=63012149-ae4f-4713-b4f3-d48f53efeebd&device_id=480f1180-8e7a-48e4-85d2-4194dc82ec55&device_id=8b6fb21a-ecca-4a32-a93c-144c0c86196a&device_id=eb6c06ad-6b12-4fa8-87c4-0ea5fe43ab6f&device_id=db311029-9dc4-46bd-bc3b-c7dc6e35725c&device_id=b3cae8ce-cc30-4ce5-b17d-6a6184aebaf4&device_id=28b62b76-af05-4115-9c06-3feb14c18e33&device_id=f90b94f7-72b2-47e1-883a-ed465c7aa9b3&device_id=874eb6b8-fce6-4c19-a402-026eb93c0399&device_id=2b99b7a0-5a31-4a97-a786-84eede30e638&device_id=04658867-e7d4-4564-9b8d-25b520fabda7&device_id=ed9832ef-9d59-416f-96eb-4d9774d4ae20&device_id=3f72b077-069a-4c5b-8f29-0c48b5c15e0f&device_id=dd0104c4-d6ae-41ac-b403-2a08742abbca&device_id=9958306f-2dd1-4835-a4d1-dc1cf19a890c&device_id=d0d6e514-796c-4a97-a9b5-e915351ae83d&device_id=afa9834a-fea3-45f1-a5b1-7b1e4464f477&device_id=997d7701-f769-4df9-8029-164b808a7ed1&device_id=b617e04b-b08b-4408-9541-9047d410d10c&device_id=ca2d692d-759c-4184-a7bd-ef83465f64e6&device_id=79824d01-2cb1-4b05-b881-2cf674ca2df4&device_id=2a1f8813-45e3-48ad-a4cb-c372f3153cde&device_id=6833cda4-3684-44a3-bfac-cf1a2bf9f99d&device_id=00529586-6b29-468b-845f-99d8e340552c&device_id=da93586d-521c-46b9-bc54-46ca5692b349&device_id=196196e2-2f1f-41fb-b5fc-da5171967c1f&device_id=d655f018-6573-4eff-ba2d-312143e71370&device_id=df0b6b2d-4b73-4486-b897-9ce29d15fe43&device_id=f63fb2ad-0158-40b2-9a9a-f68e984c2fff HTTP/1.1
User-Agent: curl/7.58.0
Accept: */*
cache-control: no-cache
postman-token: 40621dc3-a193-f668-e83b-14460c14f124
x-auth-token: gAAAAABfq12K4M94Yhwq7_gV96Lf5wMSL6BoVS6AXFJg67mCCoW06r_o-jSTP-S6IX34WHwUw60ikf0UrPz30eMtlfloXasW9rX1IMmkTYH0s0ypKfsop6fFXEc0FnZdacQ-48erdiIJ_9iMfmW4Zkuygi8mKAvI9E1ELmPYOhkRKq8wsfvMtQQ
X-Token-Info: {"token":"gAAAAABfq12K4M94Yhwq7_gV96Lf5wMSL6BoVS6AXFJg67mCCoW06r_o-jSTP-S6IX34WHwUw60ikf0UrPz30eMtlfloXasW9rX1IMmkTYH0s0ypKfsop6fFXEc0FnZdacQ-48erdiIJ_9iMfmW4Zkuygi8mKAvI9E1ELmPYOhkRKq8wsfvMtQQ", "expireAt":"2020-11-11T15:42:02.000+0000", "user":"admin1", "userId":"5179052cda854f1da2663c600f022e81", "domainId":"default", "domainName":"Default", "projectName":"admin1", "projectId":"5ddc6d5e405f4a8b8665bf2e812284f0", "projectDomain":"null", "roles":["member","admin","reader"]}
Forwarded: proto=http;host="10.213.43.171:30009";for="172.16.206.192:16850"
X-Forwarded-For: 172.16.206.192
X-Forwarded-Proto: http
X-Forwarded-Port: 30009
X-Forwarded-Host: 10.213.43.171:30009
host: portmanager-service.default.svc.cluster.local:9006
content-lengt]
2020-11-11 05:30:27.541 DEBUG 1 --- [io-8080-exec-10] o.apache.coyote.http11.Http11Processor : Error parsing HTTP request header
java.lang.IllegalArgumentException: Request header is too large
at org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:741) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.coyote.http11.Http11InputBuffer.parseHeader(Http11InputBuffer.java:824) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.coyote.http11.Http11InputBuffer.parseHeaders(Http11InputBuffer.java:584) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:283) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1594) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.33.jar!/:9.0.33]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
2020-11-11 05:30:27.541 DEBUG 1 --- [io-8080-exec-10] o.apache.catalina.valves.RemoteIpValve : Host value [10.213.43.171:30009] in HTTP header [X-Forwarded-Host] included a port number which will be ignored
2020-11-11 05:30:27.541 DEBUG 1 --- [io-8080-exec-10] o.apache.catalina.valves.RemoteIpValve : Incoming request /project/0c664c24649c4605aba77246d4b0c63f/ports with originalRemoteAddr [172.16.109.168], originalRemoteHost=[172.16.109.168], originalSecure=[false], originalScheme=[http], originalServerName=[portmanager-service.default.svc.cluster.local], originalServerPort=[9006] will be seen as newRemoteAddr=[172.16.206.192], newRemoteHost=[172.16.206.192], newSecure=[false], newScheme=[http], newServerName=[10.213.43.171], newServerPort=[30009]
2020-11-11 05:30:27.542 DEBUG 1 --- [io-8080-exec-10] o.apache.coyote.http11.Http11Processor : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@745327ce:org.apache.tomcat.util.net.NioChannel@7046c733:java.nio.channels.SocketChannel[connected local=portmanager-748bcfbfb9-4f9r6/172.16.206.235:8080 remote=/172.16.109.168:46552]], Status in: [OPEN_READ], State out: [CLOSED]
Setup API GW deployed in K8s, no issue found in local docker container or local process.
Issue From time to time, it returns 400 Bad Request for large payload of PORT LIST API. In the following curl command, if we remove one of the header field, like "cache-control" or "postman-token", or remove a few "device_id" in the URL, it works.
Input payload
HTTP Response
API GW Error Logs
Note that in the recent merged PR #465, we've increased the header size and initial line length, so this may not be not an app issue.