search

spark1security / n0s1

Secret Scanner for Slack, Jira, Confluence, Asana, Wrike and Linear
https://spark1.us/n0s1
GNU General Public License v3.0
36 stars 11 forks source link

Handle error: `JiraError HTTP 400` #20

Closed jarp0l closed 3 weeks ago

jarp0l commented 1 month ago

Is your feature request related to a problem? Please describe. While using the jira_scan command to scan Jira projects/issues, some projects only allowed me to view information like project name and project lead but not issues. So this caused the tool to not proceed to scanning the rest of the projects, and throwing JiraError HTTP 400 continuously.

Part of the error log:

JiraError HTTP 400 url: https://abcd.atlassian.net/rest/api/2/search?jql=project+%3D+%27XY%27&startAt=0&validateQuery=True&fields=%2Aall&maxResults=50
    text: The value 'XY' does not exist for the field 'project'.

    response headers = {'Date': 'Tue, 02 Jul 2024 11:37:32 GMT', 'Content-Type': 'application/json;charset=UTF-8', 'Server': 'AtlassianEdge', 'Timing-Allow-Origin': '*', 'X-Arequestid': 'REDACTED', 'X-Aaccountid': 'REDACTED', 'Cache-Control': 'no-cache, no-store, no-transform', 'X-Content-Type-Options': 'nosniff', 'X-Xss-Protection': '1; mode=block', 'Atl-Traceid': 'REDACTED', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Report-To': '{"endpoints": [{"url": "https://dy8abcen1416s.cloudfront.net"}], "group": "endpoint-1", "include_subdomains": true, "max_age": 600}', 'Nel': '{"failure_fraction": 0.001, "include_subdomains": true, "max_age": 600, "report_to": "endpoint-1"}', 'Transfer-Encoding': 'chunked'}
    response text = {"errorMessages":["The value 'XY' does not exist for the field 'project'."],"warningMessages":[]} client.search_issues(project = 'XY', startAt=0, maxResults=50)
JiraError HTTP 400 url: https://abcd.atlassian.net/rest/api/2/search?jql=project+%3D+%27XY%27&startAt=0&validateQuery=True&fields=%2Aall&maxResults=50
    text: The value 'XY' does not exist for the field 'project'.

    response headers = {'Date': 'Tue, 02 Jul 2024 11:37:34 GMT', 'Content-Type': 'application/json;charset=UTF-8', 'Server': 'AtlassianEdge', 'Timing-Allow-Origin': '*', 'X-Arequestid': 'REDACTED', 'X-Aaccountid': 'REDACTED', 'Cache-Control': 'no-cache, no-store, no-transform', 'X-Content-Type-Options': 'nosniff', 'X-Xss-Protection': '1; mode=block', 'Atl-Traceid': 'REDACTED', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Report-To': '{"endpoints": [{"url": "https://dy8abcen1416s.cloudfront.net"}], "group": "endpoint-1", "include_subdomains": true, "max_age": 600}', 'Nel': '{"failure_fraction": 0.001, "include_subdomains": true, "max_age": 600, "report_to": "endpoint-1"}', 'Transfer-Encoding': 'chunked'}
    response text = {"errorMessages":["The value 'XY' does not exist for the field 'project'."],"warningMessages":[]} client.search_issues(project = 'XY', startAt=0, maxResults=50)
JiraError HTTP 400 url: https://abcd.atlassian.net/rest/api/2/search?jql=project+%3D+%27XY%27&startAt=0&validateQuery=True&fields=%2Aall&maxResults=50
    text: The value 'XY' does not exist for the field 'project'.

    response headers = {'Date': 'Tue, 02 Jul 2024 11:37:35 GMT', 'Content-Type': 'application/json;charset=UTF-8', 'Server': 'AtlassianEdge', 'Timing-Allow-Origin': '*', 'X-Arequestid': 'REDACTED', 'X-Aaccountid': 'REDACTED', 'Cache-Control': 'no-cache, no-store, no-transform', 'X-Content-Type-Options': 'nosniff', 'X-Xss-Protection': '1; mode=block', 'Atl-Traceid': 'REDACTED', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Report-To': '{"endpoints": [{"url": "https://dy8abcen1416s.cloudfront.net"}], "group": "endpoint-1", "include_subdomains": true, "max_age": 600}', 'Nel': '{"failure_fraction": 0.001, "includpenkvv6s.cloudfront.net"}], "group": "endpoint-1", "include_subdomains": true, "max_age": 600}', 'Nel': '{"failure_fraction": 0.001, "include_subdomains": true, "max_age": 600, "report_to": "endpoint-1"}', 'Transfer-Encoding': 'chunked'}
    response text = {"errorMessages":["The value 'XY' does not exist for the field 'project'."],"warningMessages":[]} client.search_issues(project = 'XY', startAt=0, maxResults=50)
^CTraceback (most recent call last):
  File "/Users/user/tmp/.venv/lib/python3.12/site-packages/n0s1/controllers/jira_controller.py", line 87, in get_data
    issues = self._client.search_issues(ql, startAt=issue_start, maxResults=limit)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/tmp/.venv/lib/python3.12/site-packages/jira/client.py", line 3557, in search_issues
    issues = self._fetch_pages(
             ^^^^^^^^^^^^^^^^^^
  File "/Users/user/tmp/.venv/lib/python3.12/site-packages/jira/client.py", line 817, in _fetch_pages
    resource = self._get_json(
               ^^^^^^^^^^^^^^^
  File "/Users/user/tmp/.venv/lib/python3.12/site-packages/jira/client.py", line 4358, in _get_json
    else self._session.get(url, params=params)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/tmp/.venv/lib/python3.12/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/tmp/.venv/lib/python3.12/site-packages/jira/resilientsession.py", line 247, in request
    elif raise_on_error(response, **processed_kwargs):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/tmp/.venv/lib/python3.12/site-packages/jira/resilientsession.py", line 72, in raise_on_error
    raise JIRAError(
jira.exceptions.JIRAError: JiraError HTTP 400 url: https://abcd.atlassian.net/rest/api/2/search?jql=project+%3D+%27XY%27&startAt=0&validateQuery=True&fields=%2Aall&maxResults=50
    text: The value 'XY' does not exist for the field 'project'.

    response headers = {'Date': 'Tue, 02 Jul 2024 11:37:35 GMT', 'Content-Type': 'application/json;charset=UTF-8', 'Server': 'AtlassianEdge', 'Timing-Allow-Origin': '*', 'X-Arequestid': 'REDACTED', 'X-Aaccountid': 'REDACTED', 'Cache-Control': 'no-cache, no-store, no-transform', 'X-Content-Type-Options': 'nosniff', 'X-Xss-Protection': '1; mode=block', 'Atl-Traceid': 'REDACTED', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Report-To': '{"endpoints": [{"url": "https://dy8abcen1416s.cloudfront.net"}], "group": "endpoint-1", "include_subdomains": true, "max_age": 600}', 'Nel': '{"failure_fraction": 0.001, "include_subdomains": true, "max_age": 600, "report_to": "endpoint-1"}', 'Transfer-Encoding': 'chunked'}
    response text = {"errorMessages":["The value 'XY' does not exist for the field 'project'."],"warningMessages":[]}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/user/tmp/.venv/bin/n0s1", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/user/tmp/.venv/lib/python3.12/site-packages/n0s1/n0s1.py", line 631, in main
    scan(regex_config, controller, scan_arguments)
  File "/Users/user/tmp/.venv/lib/python3.12/site-packages/n0s1/n0s1.py", line 410, in scan
    for ticket in controller.get_data(scan_comment, limit):
  File "/Users/user/tmp/.venv/lib/python3.12/site-packages/n0s1/controllers/jira_controller.py", line 92, in get_data
    time.sleep(1)
KeyboardInterrupt

Describe the solution you'd like Handle the JiraError exception by adding break statement to the while not issues_finished loop to skip to the next project. This will allow the tool to continue scanning other projects even if searching issues in some projects returns a 400 error (see API docs).

Here is my implementation to handle this issue in my case: https://github.com/jarp0l/n0s1/blob/c3e84b5a91310bcae5f178c0a92072f804368f18/src/n0s1/controllers/jira_controller.py#L89-L93

while not issues_finished:
    try:
        issues = self._client.search_issues(ql, startAt=issue_start, maxResults=limit)
    except JIRAError as e:
        self.log_message(f"Error while searching issues on Jira project: [{key}]. Skipping...", logging.ERROR)
        self.log_message(e)
        issues = [{}]
        break
blupants commented 1 month ago

Thanks for sharing the issue and the solution. I will include it in the next release.

jarp0l commented 1 month ago

Thanks @blupants! 🤍 This tool was exactly what we needed, it's really a great tool.

An unrelated question: Is there any plan to add a feature to let users scan specific projects instead of scanning all of them?

blupants commented 1 month ago

Thanks @blupants! 🤍 This tool was exactly what we needed, it's really a great tool.

An unrelated question: Is there any plan to add a feature to let users scan specific projects instead of scanning all of them?

Thank you for the kind words. I am glad to hear that the tool has been useful.

In a previous discussion it was suggested to add the capability to pause and resume scans. Since I couldn't confirm from Jira/Confluence APIs that the order of the scans would always be the same (not to mention that new issues, projects or workspaces could be created in between scans) I decided to put the pause/resume feature on hold.

Though, the same feature request discussion also covered adding the "--scan-target-" and "--skip-target-" capability. That way you could create a list of specific projects you want to scan. If you think that might help you, I can split the previous feature request in 2 parts:

  1. Scan/Skip target: to be included in the next release
  2. Pause/Resume: will be kept on hold (future release)

Then, assuming you want to scan Jira project "IT", "MARKETING" and "FINANCE", you would do:

n0s1 jira_scan --server "https://<YOUR_JIRA_SERVER>.atlassian.net" --scan-target-keys "project" --scan-target-names "IT,MARKETING,FINANCE"

Would this "1. Scan/Skip target" new feature cover your use case?

jarp0l commented 1 month ago

Would this "Scan/Skip target" new feature cover your use case?

Definitely, it would. But having to add *-target-keys feels a bit tedious and unnecessary. Perhaps we should only add --scan-target-names and --skip-target-names?

It's always going to be a "project" when we are talking about Jira, and it's always going to be a "space" when we are talking about Confluence, and so on. We could also add short flags, like -sc for --scan-target-names or -sk for --skip-target-names (though the short flags can be anything else that makes sense).

Pause/Resume: will be kept on hold (future release)

I'm not sure we would need a pause/resume feature so much when we can use the scan/skip target feature. I mean, it would be great if we could have this feature, but as you said, the order of scans and the number of issues, projects, etc., may not remain the same between scans.

But maybe this could work:

blupants commented 1 month ago

Would this "Scan/Skip target" new feature cover your use case?

Definitely, it would. But having to add *-target-keys feels a bit tedious and unnecessary. Perhaps we should only add --scan-target-names and --skip-target-names?

It's always going to be a "project" when we are talking about Jira, and it's always going to be a "space" when we are talking about Confluence, and so on. We could also add short flags, like -sc for --scan-target-names or -sk for --skip-target-names (though the short flags can be anything else that makes sense).

Yes, for Jira and Confluence that would work. The problem is for tools such as Asana that have a deeper hierarchy scheme Organizations->Team->Project->Tasks. Asana users might want to filter only orgs or only teams. That's why I initially thought about the *-target-keys, but to be honest I don't completely like that approach either.

Other ticketing tools that I plan to support in the future might go even deeper in the hierarchy structure and I know it will be hard to come up with a one size-fits-all approach. One of my goals is to keep n0s1 usage as simple as possible and avoid as much as we can params that are applied to only a subset of scans.

But I understand that at some point down the road It will become impossible to support all ticketing tools with the same input args because they will differ from each other. I also think it's too early though, to dry that line in the sand. For now, we should probably have a new issue specifically to discuss how we should do filtering.

Pause/Resume: will be kept on hold (future release)

I'm not sure we would need a pause/resume feature so much when we can use the scan/skip target feature. I mean, it would be great if we could have this feature, but as you said, the order of scans and the number of issues, projects, etc., may not remain the same between scans.

But maybe this could work:

  • Create a list of all the issues/projects to scan first (like taking a snapshot at a point in time),
  • Sort them in order,
  • Save the list to a file during/after the scan, mentioning what has been scanned so far and what hasn't,
  • Use that information to pause/resume scans.

That's a great idea! Maybe we could have an argument called "mapping", that would not scan anything. It would just pull the hierarchy structure to a json or something. Then, users could edit the json according to their needs, and run a "scoped" scan using that json as an input.

Something like:

n0s1 jira_scan --server "https://<YOUR_JIRA_SERVER>.atlassian.net" --map-only "hierachy.json"
Edit/customize "hierachy.json"
n0s1 jira_scan --server "https://<YOUR_JIRA_SERVER>.atlassian.net" --scope "hierachy.json"

I suggest closing this issue so we can move on with release 1.0.19 and include the keyboard interrupt and the Jira/Confluence errors to it. I will also create a new issue specific for the "Filtering Scans" feature request, so we can get more inputs from the open source community. And once we all agree we have a good solution for filtering, we can include it in a future release.

Any thoughts?

jarp0l commented 3 weeks ago

Sorry for replying late.

Maybe we could have an argument called "mapping", that would not scan anything. It would just pull the hierarchy structure to a json or something. Then, users could edit the json according to their needs, and run a "scoped" scan using that json as an input.

Something like: ...

That would be a good idea I think.

Yeah, let's create a new issue and get feedback from the community.

Meanwhile, closing this issue for #21.