Tests fail after installation

[arfe1012]

Hey I've installed AIL and everything seems to be running fine. However, when I run the tests with ./LAUNCH.sh -t I get the following result. Is this a bug inside the test module or is there an issue with the installation?

copying /home/xxx/ail-framework/samples/2021/01/01/keys_certificat_sample.gz -> /home/xxx/ail-framework/PASTES/tests/2021/01/01 copying /home/xxx/ail-framework/samples/2021/01/01/onion.gz -> /home/xxx/ail-framework/PASTES/tests/2021/01/01 copying /home/xxx/ail-framework/samples/2021/01/01/keys.gz -> /home/xxx/ail-framework/PASTES/tests/2021/01/01 copying /home/xxx/ail-framework/samples/2021/01/01/domain_classifier.gz -> /home/xxx/ail-framework/PASTES/tests/2021/01/01 copying /home/xxx/ail-framework/samples/2021/01/01/categ.gz -> /home/xxx/ail-framework/PASTES/tests/2021/01/01 copying /home/xxx/ail-framework/samples/2021/01/01/credit_cards.gz -> /home/xxx/ail-framework/PASTES/tests/2021/01/01 copying /home/xxx/ail-framework/samples/2021/01/01/api_keys.gz -> /home/xxx/ail-framework/PASTES/tests/2021/01/01

.Module ApiKey initialized found google api key: ApiKey;tests;20210101;api_keys.gz; found AWS key: ApiKey;tests;20210101;api_keys.gz; found AWS secret key .2 CreditCards 1 Mail 1 Onion 1 Urls 2 Credential 1 Cve .Module CreditCards initialized 341039324930797 6011613905509166 5492981206527330 3547151714018657 4024007132849695 CreditCard;tests;20210101;credit_cards.gz;Checked 5 valid number(s);item::tests/2021/01/01/credit_cards.gz .Module: DomClassifier Launched ['foo.be']

set() FModule Global initialized tests/2021/01/01/global.gz File already exist /home/xxx/ail-framework/PASTES/tests/2021/01/01/global.gz File already exist /home/xxx/ail-framework/PASTES/tests/2021/01/01/global.gz ignore duplicated file /home/xxx/ail-framework/PASTES/tests/2021/01/01/global.gz File already exist /home/xxx/ail-framework/PASTES/tests/2021/01/01/global.gz File already exist /home/xxx/ail-framework/PASTES/tests/2021/01/01/global.gz tests/2021/01/01/global_831875da824fc86ab5cc0e835755b520.gz ..Module: Onion Launched http://eswpccgr5xyovsahffkehgleqthrasfpfdblwbs4lstd345dwq5qumqd.onion invalid.onion www.facebookcorewwwi.onion Onion;tests;20210101;onion.gz;Detected 2 .onion(s);item::tests/2021/01/01/onion.gz .Module Telegram initialized .

---

FAIL: test_module (test_modules.TestModuleDomClassifier.test_module)

Traceback (most recent call last): File "/home/xxx/ail-framework/tests/test_modules.py", line 105, in test_module self.assertTrue(len(result)) AssertionError: 0 is not true

---

Ran 9 tests in 5.905s

FAILED (failures=1)

Name Stmts Miss Cover

---

bin/AIL_Init.py 15 15 0% bin/Update-conf.py 58 58 0% bin/Update.py 343 343 0% bin/lib/ConfigLoader.py 63 12 81% bin/lib/Duplicate.py 73 45 38% bin/lib/Investigations.py 322 240 25% bin/lib/Language.py 273 220 19% bin/lib/Tag.py 1073 885 18% bin/lib/Tracker.py 1321 1107 16% bin/lib/Users.py 187 133 29% bin/lib/ail_core.py 131 98 25% bin/lib/ail_logger.py 20 1 95% bin/lib/ail_queues.py 281 181 36% bin/lib/btc_ail.py 45 35 22% bin/lib/chats_viewer.py 452 374 17% bin/lib/correlations_engine.py 113 71 37% bin/lib/crawlers.py 1422 1116 22% bin/lib/d4.py 57 39 32% bin/lib/data_retention_engine.py 58 24 59% bin/lib/exceptions.py 15 2 87% bin/lib/item_basic.py 152 102 33% bin/lib/objects/ChatSubChannels.py 64 35 45% bin/lib/objects/ChatThreads.py 63 39 38% bin/lib/objects/Chats.py 100 66 34% bin/lib/objects/CookiesNames.py 69 43 38% bin/lib/objects/CryptoCurrencies.py 139 109 22% bin/lib/objects/Cves.py 108 76 30% bin/lib/objects/Decodeds.py 441 360 18% bin/lib/objects/Domains.py 542 450 17% bin/lib/objects/Etags.py 69 43 38% bin/lib/objects/Favicons.py 95 64 33% bin/lib/objects/FilesNames.py 52 29 44% bin/lib/objects/HHHashs.py 84 56 33% bin/lib/objects/Images.py 105 69 34% bin/lib/objects/Items.py 496 381 23% bin/lib/objects/Messages.py 221 170 23% bin/lib/objects/Ocrs.py 239 188 21% bin/lib/objects/Pgps.py 99 74 25% bin/lib/objects/Screenshots.py 108 75 31% bin/lib/objects/Titles.py 65 40 38% bin/lib/objects/Usernames.py 84 61 27% bin/lib/objects/UsersAccount.py 140 96 31% bin/lib/objects/abstract_chat_object.py 233 174 25% bin/lib/objects/abstract_daterange_object.py 201 151 25% bin/lib/objects/abstract_object.py 215 105 51% bin/lib/objects/abstract_subtype_object.py 146 109 25% bin/lib/objects/ail_objects.py 411 329 20% bin/lib/regex_helper.py 162 122 25% bin/lib/relationships_engine.py 68 51 25% bin/lib/telegram.py 53 39 26% bin/lib/timeline_engine.py 114 93 18% bin/modules/ApiKey.py 47 2 96% bin/modules/Categ.py 49 6 88% bin/modules/Credential.py 75 75 0% bin/modules/CreditCards.py 54 10 81% bin/modules/Cryptocurrencies.py 48 48 0% bin/modules/CveModule.py 32 32 0% bin/modules/Decoder.py 85 85 0% bin/modules/DomClassifier.py 56 16 71% bin/modules/Duplicates.py 51 51 0% bin/modules/Exif.py 32 32 0% bin/modules/Global.py 119 34 71% bin/modules/Hosts.py 32 32 0% bin/modules/IPAddress.py 53 53 0% bin/modules/Iban.py 63 63 0% bin/modules/Indexer.py 91 91 0% bin/modules/Keys.py 99 45 55% bin/modules/Languages.py 21 21 0% bin/modules/LibInjection.py 44 44 0% bin/modules/MISP_Thehive_Auto_Push.py 46 46 0% bin/modules/Mail.py 110 110 0% bin/modules/Mixer.py 90 90 0% bin/modules/ModuleStats.py 24 24 0% bin/modules/OcrExtractor.py 81 81 0% bin/modules/Onion.py 62 15 76% bin/modules/Pasties.py 93 93 0% bin/modules/PgpDump.py 167 167 0% bin/modules/Phone.py 30 30 0% bin/modules/SQLInjectionDetection.py 36 36 0% bin/modules/SentimentAnalysis.py 107 107 0% bin/modules/SubmitPaste.py 186 186 0% bin/modules/Tags.py 19 19 0% bin/modules/Telegram.py 61 29 52% bin/modules/TemplateModule.py 15 15 0% bin/modules/Tools.py 35 35 0% bin/modules/Urls.py 42 42 0% bin/modules/init.py 0 0 100% bin/modules/abstract_module.py 111 50 55% bin/packages/Date.py 202 165 18% bin/packages/Hash.py 30 30 0% bin/packages/Import_helper.py 55 55 0% bin/packages/init.py 0 0 100% bin/packages/git_status.py 132 115 13% bin/packages/lib_refine.py 96 77 20% bin/packages/lib_words.py 51 51 0% bin/update-background.py 50 50 0%

---

TOTAL 14442 11256 22%

[Terrtia]

Hi @arfe1012 ! It seem like it was an issue with your DNS server configuration. Let me know if you have any issue with the DomClassifier module

[arfe1012]

Hello @Terrtia! AIL was setup in an environment, where a proxy is needed to connect to the internet. Everything seemed fine with the crawling and the analysis of the crawling-data. You're right though! the DomClassifier was a huuge bottleneck, so it might not be working correctly. I set the proxy globally and everything else worked perfectly with that. Would I need to set the proxy manually somewhere else for the DomClassifier module? I didn't look deeper into the problem but just noticed that the processing of a single item in the queue took around 5 seconds for the DomClassifier.

Maybe this information helps you. Thanks for your great work!