Add web OSINT

[hasninbl]

Dear Taranis-NG Team, First of all, thank you for this platform - Taranis-NG. Secondly, I want to add OSINT source web, https://www.cvedetails.com/cisa-known-exploited-vulnerabilities/kev-1.html but I didn't achieve the collect of the information .

[milankowww]

Hello, try the attached file securityscorecard-cisa-known-exploited.json or manually add the source using the following selectors:

• starting URL: https://www.cvedetails.com/cisa-known-exploited-vulnerabilities/kev-1.html
• landing page
  • next page button: xpath://*[@id=\"pagingt\"]/a[i[contains(@class, \"fa-chevron-right\")]]
  • article link: css:h3 > a
• article page
  • title: css:.page-header
  • short description: css:#cvedetailssummary
  • full text: css:main or pick some other part that you wish to crawl
  • published: xpath://div[span[contains(@class, \"ssc-text-secondary\") and contains(text(), \"Published\")]]

[hasninbl]

Hello, I imported the attached file, but the Taranis-NG did not display any information. for the second way manually, can you explain more:

thank you

[hsninbil]

Dear @milankowww ,

you find below the logs of the error: Requesting title page: https:// https://www.cvedetails.com/cisa-known-exploited-vulnerabilities/kev-1.html collectors_1 | [2024-03-28 12:57:53 +0100] [810] [DEBUG] [None] COLLECTOR web 'SecurityScorecard CVEdetails - Known Exploited Vulnerabilities': Error obtaining title page collectors_1 | [2024-03-28 12:57:55 +0100] [1] [ERROR] Worker (pid:1715) was sent SIGTERM! collectors_1 | [2024-03-28 12:57:55 +0100] [810] [DEBUG] [None] Traceback (most recent call last): collectors_1 | File "/app/collectors/web_collector.py", line 495, in __browse_title_page collectors_1 | browser.get(index_url) collectors_1 | File "/usr/local/lib/python3.12/site-packages/selenium/webdriver/remote/webdriver.py", line 356, in get collectors_1 | self.execute(Command.GET, {"url": url}) collectors_1 | File "/usr/local/lib/python3.12/site-packages/selenium/webdriver/remote/webdriver.py", line 347, in execute collectors_1 | self.error_handler.check_response(response) collectors_1 | File "/usr/local/lib/python3.12/site-packages/selenium/webdriver/remote/errorhandler.py", line 229, in check_response collectors_1 | raise exception_class(message, screen, stacktrace) collectors_1 | selenium.common.exceptions.InvalidArgumentException: Message: Malformed URL: URL constructor: https:// https://www.cvedetails.com/cisa-known-exploited-vulnerabilities/kev-1.html is not a valid URL. collectors_1 | Stacktrace: collectors_1 | RemoteError@chrome://remote/content/shared/RemoteError.sys.mjs:8:8 collectors_1 | WebDriverError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:192:5 collectors_1 | InvalidArgumentError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:383:5 collectors_1 | GeckoDriver.prototype.navigateTo@chrome://remote/content/marionette/driver.sys.mjs:837:11

Can you help me for fix it ? and if I have source json how can I add it i.e. https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities_schema.json ? Thank you in advance

[hasninbl]

@milankowww any advise about this request?

[Progress1]

It looks you have put wrong link: Requesting title page: https:// https://www.cvedetails.com/cisa-known-exploited-vulnerabilities/kev-1.html Do you see https:// two times? Check this field.. Maybe you put sting divided with enter? (delete field, edit and save link again)