The JSON data still cut even we set (-cl) flag for char-limit in custom webhook

ph-hitachi commented 4 weeks ago

Notify version:

1.0.6

Current Behavior:

even we set the char-limit the json output still cut on custom webhook

Expected Behavior:

the full json output will be sent to webhook

Steps To Reproduce:

nuclei -t dast/vulnerabilities -dast -j -silent | notify -id webhook -cl 500000

Anything else:

Config:

custom:
  - id: webhook
    custom_webhook_url: http://7iqcjx7fpxndjc1lztepvuvj6ac10rog.oastify.com/
    custom_method: GET
    custom_format: '{"text":{{data}} }'
    custom_headers:
      Content-Type: application/json
      X-Api-Key: XXXXX

### Tasks

RamanaReddy0M commented 3 weeks ago

@ph-hitachi can't reproduce the issue. nuclei -t dast/vulnerabilities -dast -j -silent this jsonl output length might be > 500000 charecters. I tried sending > 700000 characters line and it's working fine.

✗ cat data.txt | wc -c
  775170

$ notify -i data.txt -id webhook -cl 1000000 -proxy http://localhost:8080 Screenshot 2024-08-22 at 11 42 03 AM

try doing nuclei -t dast/vulnerabilities -dast -j -silent | wc -c

ph-hitachi commented 3 weeks ago

Hi,

Is there any chance to remove the character limitation? Like cl -1 ?

Get Outlook for Androidhttps://aka.ms/AAb9ysg

From: Ramana Reddy @.> Sent: Thursday, August 22, 2024 2:29:23 PM To: projectdiscovery/notify @.> Cc: Justin Lee @.>; Mention @.> Subject: Re: [projectdiscovery/notify] The JSON data still cut even we set (-cl) flag for char-limit in custom webhook (Issue #412)

@ph-hitachihttps://github.com/ph-hitachi can't reproduce the issue. your nuclei -t dast/vulnerabilities -dast -j -silent this jsonl output length might be > 500000 charecters. I tried sending > 700000 characters line and it's working fine.

✗ cat data.txt | wc -c 775170

$ notify -i data.txt -id webhook -cl 1000000 -proxy http://localhost:8080 Screenshot.2024-08-22.at.11.42.03.AM.png (view on web)https://github.com/user-attachments/assets/0e748101-ba18-43d6-bd66-5fd3abe8caee

try doing nuclei -t dast/vulnerabilities -dast -j -silent | wc -c

— Reply to this email directly, view it on GitHubhttps://github.com/projectdiscovery/notify/issues/412#issuecomment-2303883529, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AV5MC3GQDKZNEJV5YL7CV3TZSWAMHAVCNFSM6AAAAABMUJZP32VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBTHA4DGNJSHE. You are receiving this because you were mentioned.Message ID: @.***>

RamanaReddy0M commented 3 weeks ago

cl -1 doesn't remove character limit. you can use bulk option.

nuclei -t dast/vulnerabilities -dast -j -silent | notify -id webhook -cl 500000 -bulk

ph-hitachi commented 3 weeks ago

Unfortunately, My goal is to immediately distribute the response example, each valid domain results will be distributed to job server to process immediately without waiting the full results (bulk) for faster results is this possible?

Get Outlook for Androidhttps://aka.ms/AAb9ysg

From: Ramana Reddy @.> Sent: Thursday, August 22, 2024 7:56:43 PM To: projectdiscovery/notify @.> Cc: Justin Lee @.>; Mention @.> Subject: Re: [projectdiscovery/notify] The JSON data still cut even we set (-cl) flag for char-limit in custom webhook (Issue #412)

cl -1 doesn't remove character limit. you can use bulk option.

nuclei -t dast/vulnerabilities -dast -j -silent | notify -id webhook -cl 500000 -bulk

— Reply to this email directly, view it on GitHubhttps://github.com/projectdiscovery/notify/issues/412#issuecomment-2304486578, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AV5MC3HF7T77YNYIIMSW76TZSXGXXAVCNFSM6AAAAABMUJZP32VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBUGQ4DMNJXHA. You are receiving this because you were mentioned.Message ID: @.***>

ehsandeep commented 3 weeks ago

@ph-hitachi those limits are in place because of the providers limits, if we remove the limits but it's going to fail because those providers have limits in place and will reject the request when exceeded.

ph-hitachi commented 3 weeks ago

Limits are good options and layers, we don't need to remove that options but can look forward for additional options to remove/disabled limits for custom providers?

We should have default limits and also options to set/configure for max and even disabled the character limits

Get Outlook for Androidhttps://aka.ms/AAb9ysg

From: Sandeep Singh @.> Sent: Thursday, August 22, 2024 11:24:27 PM To: projectdiscovery/notify @.> Cc: Justin Lee @.>; Mention @.> Subject: Re: [projectdiscovery/notify] The JSON data still cut even we set (-cl) flag for char-limit in custom webhook (Issue #412)

@ph-hitachihttps://github.com/ph-hitachi those limits are in place because of the providers limits, if we remove the limits but it's going to fail because those providers have limits in place and will reject the request when exceeded.

— Reply to this email directly, view it on GitHubhttps://github.com/projectdiscovery/notify/issues/412#issuecomment-2304997843, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AV5MC3EBT4FQQ2FOIOOWTE3ZSX7CXAVCNFSM6AAAAABMUJZP32VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBUHE4TOOBUGM. You are receiving this because you were mentioned.Message ID: @.***>

projectdiscovery / notify