[RSS Feed] Error 403 on accessible public feeds

[Lhorus6]

Description

Some RSS feeds return errors 403 even though the links are public and accessible.

Examples:

• https://dailydarkweb.net/feed/
• https://cybersecurity.att.com/site/blog-all-rss
• https://www.securityweek.com/feed/

Environment

OCTI 6.3.6

Reproducible Steps

Steps to create the smallest reproducible scenario:

1. Create an RSS Feed with one of the examples in the description
2. Wait a bit and look at the platform logs.

Expected Output

Ingestion of the RSS feed

Actual Output

Error 403

Screenshots

[khalidelborai]

Same issue here.

[SamuelHassine]

Maybe a problem of user agent? @romain-filigran @nino-filigran maybe critical?

[romain-filigran]

@SamuelHassine: Not only a user-agent issue from my investigation, more complicated depending on the RSS source. Need to test which one is working with an external RSS tool to identify the problem.

[biastogit]

it seems this issue appeared on my instance 3 days ago after adding the RSS darkreading feed. I started disabling all the RSS feeds creating errors, without success. Also tried to delete the darkreading reports and have multiple update indexing fail errors (UI and logs)

[SamuelHassine]

Any news @romain-filigran @nino-filigran ?

[nino-filigran]

None @SamuelHassine we've increased priority to ensure it's looked over by devs. So far it seems sepcific to some RSS feeds & therefore the solution not straightforward, but we need to investigate.