search

CybercentreCanada / assemblyline-service-badlist

Assemblyline 4 service for badlisting network and file features
https://cybercentrecanada.github.io/assemblyline4_docs/
MIT License
1 stars 3 forks source link

Badlist Signatures updater issue #24

Closed githule closed 2 months ago

githule commented 3 months ago

Hello,

Since my Asssembly Line setup is on, I wasnt able to pull signatures from badlist service. I am using only default sources with following configuration :

In the logs I have following error "MGet returned a document without any data" Other services / sandboxes communicate well via proxy. Accessing source url with a browser returned appropriate content. Please note it is not easy to identify which source is failing that update.

Thank you for your helps on that issue.

githule commented 3 months ago

Also it seems that a failing source is failing all sources from updating => resulting in no signatures at all. Would like to have more logs/details concerning this issue. All I have for now. Dont hesitate to show me debug mode or things like that :)

cccs-rs commented 3 months ago

You should be able to see which sources have failed in the UI under Update Sources

githule commented 3 months ago

Thank you for your answer. Yes, I see that. But the issue is that I cant tell why. Provided sources such as urlhaus or blackbook do not update and report no error in the GUI except "Last Successful Update: 54 years ago". I am looking in a way to have more verbose reports in case of a faillure. Other services/updaters seems to be easier to debug.

cccs-rs commented 3 months ago

There should be logs from the updater container that should indicate why it failed per source.

githule commented 3 months ago

Just forced an update to latest stable version of AL and fixed an obsolete proxy config in the compose file affecting updater container, now some sources are updating but dont know well to which root cause it is bound. I will check in next days if its definitely solved. Thank for your time

githule commented 3 months ago

I come back with some news, in my logs I found following entry "Looks like index BADLIST is not ready yet." Is it an issue with index creation during init of the platform ? How can I make sure to fix "maybe missing" indexes "safely" without having to rebuild the platform from scratch ? If you have some way to handle that it will be a great news. :)

githule commented 2 months ago

Maybe something went wrong during setup of the index in the elasticsearch. I built it without the elk monitoring stack so I might be a little limited in term of finding the root cause. Currently disabled Badlist service because that issue systematically leads to timeout during analysis.

githule commented 2 months ago

Went a little deeper in elastic debug, seems my shards were corrupted but only for the badlist index. I am just sad it didn't throw an error on the GUI and entered in an silent error loop. I will improve my elastic skills before the next issue :). I mark as fixed. Thank you for your time.