search

CrowdStrike / MISP-tools

Import CrowdStrike Threat Intelligence into your instance of MISP
MIT License
37 stars 9 forks source link

OverflowError: string longer than 2147483647 bytes #102

Closed packet-rat closed 6 months ago

packet-rat commented 1 year ago

230225-2300

python3 misp_import.py --fullmonty -d -v -p -nb

[2023-02-25 22:40:39,751] INFO     misp_tools    MISP Import for CrowdStrike Threat Intelligence v0.6.8
[2023-02-25 22:40:39,751] INFO     config  CHECK CONFIG
[2023-02-25 22:40:39,752] DEBUG    config  client_id                                   value redacted, check config file
[2023-02-25 22:40:39,752] DEBUG    config  client_secret                               value redacted, check config file
[2023-02-25 22:40:39,752] DEBUG    config  crowdstrike_url                             US1
[2023-02-25 22:40:39,752] DEBUG    config  api_request_max                             5000
[2023-02-25 22:40:39,752] DEBUG    config  api_enable_ssl                              True
[2023-02-25 22:40:39,752] DEBUG    config  reports_timestamp_filename                  lastReportsUpdate.dat
[2023-02-25 22:40:39,752] DEBUG    config  indicators_timestamp_filename               lastIndicatorsUpdate.dat
[2023-02-25 22:40:39,752] DEBUG    config  actors_timestamp_filename                   lastActorsUpdate.dat
[2023-02-25 22:40:39,752] DEBUG    config  init_reports_days_before                    365
[2023-02-25 22:40:39,752] DEBUG    config  init_indicators_minutes_before              20220
[2023-02-25 22:40:39,752] DEBUG    config  init_actors_days_before                     365
[2023-02-25 22:40:39,752] DEBUG    config  reports_unique_tag                          CrowdStrike: REPORT
[2023-02-25 22:40:39,752] DEBUG    config  indicators_unique_tag                       CrowdStrike: INDICATOR
[2023-02-25 22:40:39,753] DEBUG    config  actors_unique_tag                           CrowdStrike: ADVERSARY
[2023-02-25 22:40:39,753] DEBUG    config  reports_tags                                att:source="Crowdstrike.Report"
[2023-02-25 22:40:39,753] DEBUG    config  indicators_tags                             att:source="Crowdstrike.Indicators"
[2023-02-25 22:40:39,753] DEBUG    config  actors_tags                                 att:source="Crowdstrike.Actors"
[2023-02-25 22:40:39,753] DEBUG    config  unknown_mapping                             CrowdStrike:indicator:galaxy: UNATTRIBUTED
[2023-02-25 22:40:39,753] DEBUG    config  unattributed_title                          Unattributed indicators:
[2023-02-25 22:40:39,753] DEBUG    config  indicator_type_title                        Indicator Type:
[2023-02-25 22:40:39,753] DEBUG    config  malware_family_title                        Malware Family:
[2023-02-25 22:40:39,753] DEBUG    config  log_duplicates_as_sightings                 True
[2023-02-25 22:40:39,753] DEBUG    config  misp_url                                    https://3samisp/
[2023-02-25 22:40:39,753] DEBUG    config  misp_auth_key                               value redacted, check config file
[2023-02-25 22:40:39,753] DEBUG    config  crowdstrike_org_uuid                        ca4f4b5d-db04-4a5e-a6de-e60636dc01be
[2023-02-25 22:40:39,753] DEBUG    config  misp_enable_ssl                             False
[2023-02-25 22:40:39,753] WARNING  config  misp_enable_ssl                             SSL is disabled for MISP API requests
[2023-02-25 22:40:39,753] DEBUG    config  ind_attribute_batch_size                    2500
[2023-02-25 22:40:39,753] DEBUG    config  event_save_memory_refresh_interval          180
[2023-02-25 22:40:39,753] DEBUG    config  max_threads                                 16
[2023-02-25 22:40:39,753] DEBUG    config  miss_track_file                             no_galaxy_mapping.log
[2023-02-25 22:40:39,753] DEBUG    config  galaxies_map_file                           galaxy.ini
[2023-02-25 22:40:39,753] DEBUG    config  tag_unknown_galaxy_maps                     True
[2023-02-25 22:40:39,753] DEBUG    config  taxonomic_kill-chain                        True
[2023-02-25 22:40:39,753] DEBUG    config  taxonomic_information-security-data-source  True
[2023-02-25 22:40:39,753] DEBUG    config  taxonomic_type                              True
[2023-02-25 22:40:39,753] DEBUG    config  taxonomic_iep                               False
[2023-02-25 22:40:39,753] DEBUG    config  taxonomic_iep2                              True
[2023-02-25 22:40:39,753] DEBUG    config  taxonomic_iep2_version                      False
[2023-02-25 22:40:39,753] DEBUG    config  taxonomic_tlp                               True
[2023-02-25 22:40:39,753] DEBUG    config  taxonomic_workflow                          True
[2023-02-25 22:40:40,646] INFO     config  No configuration errors found (1 warning)
[2023-02-25 22:40:40,646] INFO     config  
[2023-02-25 22:40:40,646] INFO     config  ____ _  _ ____ ____ _  _ ____    ___  ____ ____ ____ ____ ___
[2023-02-25 22:40:40,646] INFO     config  |    |__| |___ |    |_/  [__     |__] |__| [__  [__  |___ |  \
[2023-02-25 22:40:40,646] INFO     config  |___ |  | |___ |___ | \_ ___]    |    |  | ___] ___] |___ |__/
[2023-02-25 22:40:40,646] INFO     config  
[2023-02-25 22:40:42,246] INFO     processor/main       BEGIN ADVERSARIES IMPORT
[2023-02-25 22:40:42,247] INFO     processor/main       Start importing CrowdStrike Adversaries as events into MISP (past 365 days).
[2023-02-25 22:40:42,247] INFO     processor/main       Retrieving all adversaries.
[2023-02-25 22:40:43,061] INFO     processor/main       Got 0 adversaries from the Crowdstrike Intel API.
[2023-02-25 22:40:43,063] INFO     processor/main       Finished importing CrowdStrike Adversaries as events into MISP.
[2023-02-25 22:40:43,064] INFO     processor/main       Completed import of adversaries into MISP in 0.82 seconds
[2023-02-25 22:40:43,064] INFO     processor/main       BEGIN REPORTS IMPORT
[2023-02-25 22:40:43,064] INFO     processor/main       Starting import of CrowdStrike Threat Intelligence reports as events (past 365 days).
[2023-02-25 22:40:43,065] INFO     processor/main       Retrieving all available report types.
[2023-02-25 22:40:43,469] INFO     processor/main       Retrieved 0 total reports from the Crowdstrike Intel API.
[2023-02-25 22:40:43,470] INFO     processor/main       Found 2433 pre-existing CrowdStrike reports within the MISP instance.
[2023-02-25 22:40:43,471] INFO     processor/main       Finished importing 0 (0 skipped) Crowdstrike Threat Intelligence reports.
[2023-02-25 22:40:43,471] INFO     processor/main       Completed import of reports into MISP in 0.40 seconds
[2023-02-25 22:40:43,472] INFO     processor/main       BEGIN INDICATORS IMPORT
[2023-02-25 22:40:43,472] INFO     processor/main       Retrieving lookup data for import of CrowdStrike indicators into MISP.
[2023-02-25 22:40:43,510] INFO     processor/main       Retrieved 24 CrowdStrike indicator type events from MISP.
[2023-02-25 22:42:17,322] INFO     processor/main       Retrieved 229 CrowdStrike indicator malware family events from MISP.
[2023-02-25 22:42:18,486] INFO     processor/thread_0   Processed Malware Family: Tofsee
[2023-02-25 22:42:19,550] INFO     processor/thread_0   Processed Malware Family: IIS7VbDropper
[2023-02-25 22:42:27,511] INFO     processor/thread_5   Processed Malware Family: XMRig
[2023-02-25 22:42:31,917] INFO     processor/thread_12  Processed Malware Family: AmadeyStealer
[2023-02-25 22:42:32,380] INFO     processor/thread_12  Processed Malware Family: DelphiCorePacker
[2023-02-25 22:42:32,725] INFO     processor/thread_12  Processed Malware Family: Nymaim
[2023-02-25 22:42:34,308] INFO     processor/thread_12  Processed Malware Family: CybergateRAT
[2023-02-25 22:42:50,625] INFO     processor/thread_12  Processed Malware Family: BokBot
[2023-02-25 22:42:51,102] INFO     processor/thread_12  Processed Malware Family: MatiexKeylogger
[2023-02-25 22:42:51,483] INFO     processor/thread_15  Processed Malware Family: STOP
[2023-02-25 22:42:53,337] INFO     processor/thread_12  Processed Malware Family: Azorult
[2023-02-25 22:42:57,115] INFO     processor/thread_6   Processed Malware Family: Kovter
[2023-02-25 22:42:57,465] INFO     processor/thread_12  Processed Malware Family: Mispadu
[2023-02-25 22:42:58,122] INFO     processor/thread_7   Processed Malware Family: DarkComet
[2023-02-25 22:42:58,875] INFO     processor/thread_7   Processed Malware Family: Golroted
[2023-02-25 22:42:59,079] INFO     processor/thread_7   Processed Malware Family: Gootkit
[2023-02-25 22:43:00,584] INFO     processor/thread_10  Processed Malware Family: XLoader
[2023-02-25 22:43:02,455] INFO     processor/thread_12  Processed Malware Family: Salityv2
[2023-02-25 22:43:02,458] INFO     processor/thread_12  Processed Malware Family: BuhtrapLoader
[2023-02-25 22:43:02,766] INFO     processor/thread_12  Processed Malware Family: PandaZeus
[2023-02-25 22:43:03,110] INFO     processor/thread_12  Processed Malware Family: Kinsing
[2023-02-25 22:43:04,351] INFO     processor/thread_10  Processed Malware Family: LockBitRansomware
[2023-02-25 22:43:06,034] INFO     processor/thread_6   Processed Malware Family: Badnews
[2023-02-25 22:43:06,981] INFO     processor/thread_12  Processed Malware Family: FormBook
[2023-02-25 22:43:07,010] INFO     processor/thread_10  Processed Malware Family: NanoCore
[2023-02-25 22:43:15,387] INFO     processor/thread_12  Processed Malware Family: Phorpiex
[2023-02-25 22:43:19,217] INFO     processor/thread_8   Processed Malware Family: PdfCaptchaLure21
[2023-02-25 22:43:19,745] INFO     processor/thread_12  Processed Malware Family: Rekram
[2023-02-25 22:43:20,025] INFO     processor/thread_10  Processed Malware Family: Warzone
[2023-02-25 22:43:20,876] INFO     processor/thread_8   Processed Malware Family: Vflooder
[2023-02-25 22:43:26,617] INFO     processor/thread_9   Processed Malware Family: Pony
[2023-02-25 22:43:26,623] INFO     processor/thread_9   Processed Malware Family: PythonInMemoryStager
[2023-02-25 22:43:26,893] INFO     processor/thread_9   Processed Malware Family: SolarBot
[2023-02-25 22:43:31,002] INFO     processor/thread_9   Processed Malware Family: Netwire
[2023-02-25 22:43:32,041] INFO     processor/thread_0   Processed Malware Family: RedLineStealer
[2023-02-25 22:43:35,047] INFO     processor/thread_7   Processed Malware Family: WanaRansomware
[2023-02-25 22:43:35,049] INFO     processor/thread_7   Processed Malware Family: MyloBot/Dropper
[2023-02-25 22:43:35,672] INFO     processor/thread_8   Processed Malware Family: AsyncRAT
[2023-02-25 22:43:35,680] INFO     processor/thread_8   Processed Malware Family: Bat2Exe
[2023-02-25 22:43:35,881] INFO     processor/thread_8   Processed Malware Family: SpyMax
[2023-02-25 22:43:36,096] INFO     processor/thread_12  Processed Malware Family: CryptBot
[2023-02-25 22:43:38,203] INFO     processor/thread_0   Processed Malware Family: MyloBot
[2023-02-25 22:43:38,208] INFO     processor/thread_0   Processed Malware Family: H1N1
[2023-02-25 22:43:38,351] INFO     processor/thread_12  Processed Malware Family: FickerStealer
[2023-02-25 22:43:38,519] INFO     processor/thread_15  Processed Malware Family: Sakula
[2023-02-25 22:43:39,930] INFO     processor/thread_7   Processed Malware Family: MyloPacker
[2023-02-25 22:43:40,149] INFO     processor/thread_7   Processed Malware Family: KopiLuwak
[2023-02-25 22:43:40,689] INFO     processor/thread_15  Processed Malware Family: ISFB
[2023-02-25 22:43:43,273] INFO     processor/thread_15  Processed Malware Family: Metasploit
[2023-02-25 22:43:43,456] INFO     processor/thread_0   Processed Malware Family: AgentTesla
[2023-02-25 22:43:49,764] INFO     processor/thread_15  Processed Malware Family: JsOutProx
[2023-02-25 22:43:49,773] INFO     processor/thread_15  Processed Malware Family: UknownRAT
[2023-02-25 22:43:50,503] INFO     processor/thread_15  Processed Malware Family: Astaroth
[2023-02-25 22:43:50,663] INFO     processor/thread_15  Processed Malware Family: Meterpreter
[2023-02-25 22:43:50,908] INFO     processor/thread_15  Processed Malware Family: PoisonIvy
[2023-02-25 22:43:52,180] INFO     processor/thread_15  Processed Malware Family: VidarStealer
[2023-02-25 22:43:52,456] INFO     processor/thread_15  Processed Malware Family: Andromeda
[2023-02-25 22:43:52,720] INFO     processor/thread_15  Processed Malware Family: Phishery
[2023-02-25 22:43:52,915] INFO     processor/thread_15  Processed Malware Family: BlueHeaven
[2023-02-25 22:43:54,066] INFO     processor/thread_10  Processed Malware Family: KpotStealer
[2023-02-25 22:43:54,692] INFO     processor/thread_10  Processed Malware Family: WideGate
[2023-02-25 22:43:54,835] INFO     processor/thread_10  Processed Malware Family: Caiman
[2023-02-25 22:43:57,564] INFO     processor/thread_10  Processed Malware Family: XORDDoS
[2023-02-25 22:44:01,324] INFO     processor/thread_0   Processed Malware Family: Qakbot
[2023-02-25 22:44:02,024] INFO     processor/thread_10  Processed Malware Family: njRATLime
[2023-02-25 22:44:02,027] INFO     processor/thread_10  Processed Malware Family: DiamondFox
[2023-02-25 22:44:02,081] INFO     processor/thread_9   Processed Malware Family: LokiBot
[2023-02-25 22:44:02,141] INFO     processor/thread_10  Processed Malware Family: Hancitor
[2023-02-25 22:44:02,471] INFO     processor/thread_0   Processed Malware Family: Nitol
[2023-02-25 22:44:02,491] INFO     processor/thread_9   Processed Malware Family: AhMyth
[2023-02-25 22:44:02,532] INFO     processor/thread_10  Processed Malware Family: Matanbuchus
[2023-02-25 22:44:02,865] INFO     processor/thread_10  Processed Malware Family: TinyLoader
[2023-02-25 22:44:03,478] INFO     processor/thread_9   Processed Malware Family: TrojanizedDocument
[2023-02-25 22:44:03,803] INFO     processor/thread_0   Processed Malware Family: Loda
[2023-02-25 22:44:03,838] INFO     processor/thread_0   Processed Malware Family: YahooStealer
[2023-02-25 22:44:04,077] INFO     processor/thread_0   Processed Malware Family: BlackEnergy
[2023-02-25 22:44:04,078] INFO     processor/thread_0   Processed Malware Family: ScreenConnect
[2023-02-25 22:44:06,655] INFO     processor/thread_0   Processed Malware Family: Emotet
[2023-02-25 22:44:08,153] INFO     processor/thread_0   Processed Malware Family: RaccoonStealer
[2023-02-25 22:44:08,157] INFO     processor/thread_0   Processed Malware Family: Rozena
[2023-02-25 22:44:08,159] INFO     processor/thread_0   Processed Malware Family: CubeCrypter
[2023-02-25 22:44:09,220] INFO     processor/thread_9   Processed Malware Family: Remcos
[2023-02-25 22:44:09,401] INFO     processor/thread_9   Processed Malware Family: Gh0stRAT
[2023-02-25 22:44:09,407] INFO     processor/thread_9   Processed Malware Family: SystemBC
[2023-02-25 22:44:09,520] INFO     processor/thread_9   Processed Malware Family: BankBotAnubis
[2023-02-25 22:44:09,673] INFO     processor/thread_9   Processed Malware Family: XtremeRAT
[2023-02-25 22:44:10,426] INFO     processor/thread_9   Processed Malware Family: PhorpiexDownloader
[2023-02-25 22:44:11,143] INFO     processor/thread_9   Processed Malware Family: Enosch
[2023-02-25 22:44:11,505] INFO     processor/thread_9   Processed Malware Family: Xworm
[2023-02-25 22:44:11,943] INFO     processor/thread_7   Processed Malware Family: Quasar
[2023-02-25 22:44:11,986] INFO     processor/thread_9   Processed Malware Family: SnakeKeylogger
[2023-02-25 22:44:12,155] INFO     processor/thread_9   Processed Malware Family: MixLoader
[2023-02-25 22:44:12,206] INFO     processor/thread_9   Processed Malware Family: MiniPythonConnectBackShell
[2023-02-25 22:44:13,284] INFO     processor/thread_9   Processed Malware Family: Necast
[2023-02-25 22:44:13,478] INFO     processor/thread_9   Processed Malware Family: HalfAndHalfDownloader
[2023-02-25 22:44:13,481] INFO     processor/thread_9   Processed Malware Family: PhishingAttachment
[2023-02-25 22:44:13,627] INFO     processor/thread_9   Processed Malware Family: PhishingLureWithInjectedTemplate
[2023-02-25 22:44:14,121] INFO     processor/thread_11  Processed Malware Family: Mofksys
[2023-02-25 22:44:14,584] INFO     processor/thread_11  Processed Malware Family: FakeBrowserUpdate
[2023-02-25 22:44:14,793] INFO     processor/thread_9   Processed Malware Family: DharmaRansomware
[2023-02-25 22:44:14,891] INFO     processor/thread_9   Processed Malware Family: BlackBasta
[2023-02-25 22:44:14,896] INFO     processor/thread_9   Processed Malware Family: Apk4cr
[2023-02-25 22:44:15,096] INFO     processor/thread_9   Processed Malware Family: FTT
[2023-02-25 22:44:15,304] INFO     processor/thread_11  Processed Malware Family: Kiron
[2023-02-25 22:44:15,405] INFO     processor/thread_11  Processed Malware Family: RM3
[2023-02-25 22:44:15,506] INFO     processor/thread_9   Processed Malware Family: Salve
[2023-02-25 22:44:15,513] INFO     processor/thread_11  Processed Malware Family: ZeusVM
[2023-02-25 22:44:15,516] INFO     processor/thread_11  Processed Malware Family: Qakbot/VNCPlugin
[2023-02-25 22:44:15,519] INFO     processor/thread_11  Processed Malware Family: Megatron
[2023-02-25 22:44:15,543] INFO     processor/thread_9   Processed Malware Family: FBILocker
[2023-02-25 22:44:15,722] INFO     processor/thread_11  Processed Malware Family: NetSupportRAT
[2023-02-25 22:44:15,775] INFO     processor/thread_11  Processed Malware Family: PhobosRansomware
[2023-02-25 22:44:15,856] INFO     processor/thread_11  Processed Malware Family: Kronos
[2023-02-25 22:44:15,927] INFO     processor/thread_9   Processed Malware Family: Dridex
[2023-02-25 22:44:15,963] INFO     processor/thread_11  Processed Malware Family: ZxShell
[2023-02-25 22:44:15,965] INFO     processor/thread_9   Processed Malware Family: EggJagger
[2023-02-25 22:44:15,968] INFO     processor/thread_9   Processed Malware Family: Ratty2
[2023-02-25 22:44:16,060] INFO     processor/thread_11  Processed Malware Family: Lampion
[2023-02-25 22:44:16,062] INFO     processor/thread_11  Processed Malware Family: Skynet
[2023-02-25 22:44:16,066] INFO     processor/thread_11  Processed Malware Family: SwaySpy
[2023-02-25 22:44:16,069] INFO     processor/thread_9   Processed Malware Family: CoreImpact
[2023-02-25 22:44:16,159] INFO     processor/thread_11  Processed Malware Family: Culebra
[2023-02-25 22:44:16,202] INFO     processor/thread_11  Processed Malware Family: CoreBot
[2023-02-25 22:44:16,272] INFO     processor/thread_11  Processed Malware Family: FlawedAmmyy
[2023-02-25 22:44:16,275] INFO     processor/thread_11  Processed Malware Family: RevengeRAT
[2023-02-25 22:44:16,277] INFO     processor/thread_11  Processed Malware Family: MatrixRansomware
[2023-02-25 22:44:16,436] INFO     processor/thread_9   Processed Malware Family: CulebraVariant
[2023-02-25 22:44:16,459] INFO     processor/thread_9   Processed Malware Family: Kronos
[2023-02-25 22:44:16,470] INFO     processor/thread_9   Processed Malware Family: PowerSploitDLL
[2023-02-25 22:44:16,570] INFO     processor/thread_9   Processed Malware Family: TrojanizedWinRMDownloader
[2023-02-25 22:44:17,186] INFO     processor/thread_11  Processed Malware Family: HawkEyeKeylogger
[2023-02-25 22:44:17,489] INFO     processor/thread_11  Processed Malware Family: NetInfoNabster
[2023-02-25 22:44:17,583] INFO     processor/thread_11  Processed Malware Family: Piccoload
[2023-02-25 22:44:17,584] INFO     processor/thread_11  Processed Malware Family: RatAttack
[2023-02-25 22:44:17,747] INFO     processor/thread_10  Processed Malware Family: USnapDownloader
[2023-02-25 22:44:17,877] INFO     processor/thread_10  Processed Malware Family: AllinOneNeo
[2023-02-25 22:44:17,880] INFO     processor/thread_10  Processed Malware Family: Mineware
[2023-02-25 22:44:17,883] INFO     processor/thread_10  Processed Malware Family: UltraVNC
[2023-02-25 22:44:18,218] INFO     processor/thread_11  Processed Malware Family: X-Agent
[2023-02-25 22:44:18,228] INFO     processor/thread_10  Processed Malware Family: Vulcanops
[2023-02-25 22:44:18,231] INFO     processor/thread_10  Processed Malware Family: MorphineRAT
[2023-02-25 22:44:18,234] INFO     processor/thread_10  Processed Malware Family: PhoenixKeylogger
[2023-02-25 22:44:18,236] INFO     processor/thread_10  Processed Malware Family: XsltAspxWebshell
[2023-02-25 22:44:18,320] INFO     processor/thread_10  Processed Malware Family: A1Lock
[2023-02-25 22:44:18,387] INFO     processor/thread_11  Processed Malware Family: DoppelDridex
[2023-02-25 22:44:18,429] INFO     processor/thread_11  Processed Malware Family: BazarLoader
[2023-02-25 22:44:18,512] INFO     processor/thread_11  Processed Malware Family: Comome
[2023-02-25 22:44:18,568] INFO     processor/thread_9   Processed Malware Family: TrickBot
[2023-02-25 22:44:18,576] INFO     processor/thread_9   Processed Malware Family: TaurusLoaderStealerModule
[2023-02-25 22:44:18,581] INFO     processor/thread_11  Processed Malware Family: NocturnalStealer
[2023-02-25 22:44:18,599] INFO     processor/thread_9   Processed Malware Family: vw0rm
[2023-02-25 22:44:18,651] INFO     processor/thread_10  Processed Malware Family: PSCrypt
[2023-02-25 22:44:18,866] INFO     processor/thread_10  Processed Malware Family: SyncroRemoteAccess
[2023-02-25 22:44:19,088] INFO     processor/thread_10  Processed Malware Family: ParasiteHTTP
[2023-02-25 22:44:19,171] INFO     processor/thread_10  Processed Malware Family: SlooPower
[2023-02-25 22:44:19,559] INFO     processor/thread_10  Processed Malware Family: Chinch
[2023-02-25 22:44:19,563] INFO     processor/thread_10  Processed Malware Family: Cutwail
[2023-02-25 22:44:19,571] INFO     processor/thread_10  Processed Malware Family: Sykipot
[2023-02-25 22:44:19,575] INFO     processor/thread_10  Processed Malware Family: BozokRAT
[2023-02-25 22:44:19,659] INFO     processor/thread_10  Processed Malware Family: Donut
[2023-02-25 22:44:19,698] INFO     processor/thread_10  Processed Malware Family: TeamSpy
[2023-02-25 22:44:19,778] INFO     processor/thread_10  Processed Malware Family: TimeStealerTriggerCobaltStrike
[2023-02-25 22:44:19,780] INFO     processor/thread_10  Processed Malware Family: VendettaBackdoor
[2023-02-25 22:44:19,904] INFO     processor/thread_10  Processed Malware Family: Prorock
[2023-02-25 22:44:19,912] INFO     processor/thread_10  Processed Malware Family: Onliner
[2023-02-25 22:44:19,988] INFO     processor/thread_10  Processed Malware Family: MasterTape
[2023-02-25 22:44:20,090] INFO     processor/thread_11  Processed Malware Family: HWorm
[2023-02-25 22:44:20,110] INFO     processor/thread_10  Processed Malware Family: TwoFaceWebShell
[2023-02-25 22:44:20,306] INFO     processor/thread_10  Processed Malware Family: Baryonyx
[2023-02-25 22:44:20,309] INFO     processor/thread_10  Processed Malware Family: LeoDocument
[2023-02-25 22:44:20,358] INFO     processor/thread_11  Processed Malware Family: PhishingShortcutLnk
[2023-02-25 22:44:20,408] INFO     processor/thread_10  Processed Malware Family: KoloGrabber
[2023-02-25 22:44:20,512] INFO     processor/thread_11  Processed Malware Family: BuildYourOwnBotnet
[2023-02-25 22:44:20,649] INFO     processor/thread_11  Processed Malware Family: BlisterLoader
[2023-02-25 22:44:20,669] INFO     processor/thread_11  Processed Malware Family: Emotet/PluginLoader
[2023-02-25 22:44:20,680] INFO     processor/thread_11  Processed Malware Family: BatLoader
[2023-02-25 22:44:21,364] INFO     processor/thread_7   Processed Malware Family: GuLoader
[2023-02-25 22:44:21,453] INFO     processor/thread_7   Processed Malware Family: Taleret
[2023-02-25 22:44:21,456] INFO     processor/thread_7   Processed Malware Family: HAVEX
[2023-02-25 22:44:21,492] INFO     processor/thread_7   Processed Malware Family: STRRAT
[2023-02-25 22:44:21,581] INFO     processor/thread_7   Processed Malware Family: liderc
[2023-02-25 22:44:21,589] INFO     processor/thread_7   Processed Malware Family: HailMary
[2023-02-25 22:44:21,674] INFO     processor/thread_7   Processed Malware Family: MedusaLocker
[2023-02-25 22:44:21,675] INFO     processor/thread_7   Processed Malware Family: Metamorfo
[2023-02-25 22:44:21,732] INFO     processor/thread_7   Processed Malware Family: Makop
[2023-02-25 22:44:21,809] INFO     processor/thread_7   Processed Malware Family: InstituteX
[2023-02-25 22:44:21,903] INFO     processor/thread_7   Processed Malware Family: WinExe
[2023-02-25 22:44:21,931] INFO     processor/thread_9   Processed Malware Family: Zloader
[2023-02-25 22:44:22,121] INFO     processor/thread_7   Processed Malware Family: Mozart
[2023-02-25 22:44:22,264] INFO     processor/thread_9   Processed Malware Family: MsmRat
[2023-02-25 22:44:22,329] INFO     processor/thread_7   Processed Malware Family: Ishmael
[2023-02-25 22:44:22,340] INFO     processor/thread_7   Processed Malware Family: Danabot
[2023-02-25 22:44:22,341] INFO     processor/thread_7   Processed Malware Family: PlugX
[2023-02-25 22:44:22,369] INFO     processor/thread_9   Processed Malware Family: TimeStealerTrigger
[2023-02-25 22:44:22,370] INFO     processor/thread_9   Processed Malware Family: SolarMarkerBackdoorInstaller
[2023-02-25 22:44:22,426] INFO     processor/thread_7   Processed Malware Family: SolarMarkerBackdoor
[2023-02-25 22:44:22,711] INFO     processor/thread_7   Processed Malware Family: CloudMensis
[2023-02-25 22:44:22,790] INFO     processor/thread_9   Processed Malware Family: SolarMarkerPowerShellLoader
[2023-02-25 22:44:22,799] INFO     processor/thread_7   Processed Malware Family: Chthonic
[2023-02-25 22:44:22,812] INFO     processor/thread_9   Processed Malware Family: EvilGnome
[2023-02-25 22:44:23,358] INFO     processor/thread_9   Processed Malware Family: EvilGnomeLinux
[2023-02-25 22:44:23,814] INFO     processor/thread_10  Processed Malware Family: ClopRansomware
[2023-02-25 22:44:23,818] INFO     processor/thread_9   Processed Malware Family: StatusSymbol
[2023-02-25 22:44:24,271] INFO     processor/thread_10  Processed Malware Family: SparkDownloader
[2023-02-25 22:44:24,407] INFO     processor/thread_10  Processed Malware Family: Proxy
[2023-02-25 22:44:24,409] INFO     processor/thread_10  Processed Malware Family: CarbonSpiderStagerDLL
[2023-02-25 22:44:24,527] INFO     processor/thread_10  Processed Malware Family: LizarStager
[2023-02-25 22:44:24,696] INFO     processor/thread_10  Processed Malware Family: RisePro
[2023-02-25 22:44:24,869] INFO     processor/thread_10  Processed Malware Family: BetaBot
[2023-02-25 22:44:24,870] INFO     processor/thread_10  Processed Malware Family: BianLianRansomware
[2023-02-25 22:44:24,958] INFO     processor/thread_10  Processed Malware Family: Ransomware
[2023-02-25 22:44:24,960] INFO     processor/thread_10  Processed Malware Family: NTSTATS
[2023-02-25 22:44:24,971] INFO     processor/thread_10  Processed Malware Family: ShinobuClipper
[2023-02-25 22:44:25,096] INFO     processor/thread_9   Processed Malware Family: SocksProxyGo
[2023-02-25 22:44:25,434] INFO     processor/thread_10  Processed Malware Family: CraP2P
[2023-02-25 22:44:25,705] INFO     processor/thread_7   Processed Malware Family: EvilGnomeWindows
[2023-02-25 22:44:36,006] INFO     processor/thread_14  Processed Malware Family: AmadeyLoader
[2023-02-25 22:44:43,121] INFO     processor/thread_0   Processed Malware Family: Magecart
[2023-02-25 22:44:47,067] INFO     processor/thread_11  Processed Malware Family: Salityv4
[2023-02-25 22:44:53,989] INFO     processor/thread_12  Processed Malware Family: Tinba
[2023-02-25 22:45:11,020] INFO     processor/thread_2   Processed Malware Family: SmokeLoader
[2023-02-25 22:45:13,435] INFO     processor/thread_4   Processed Malware Family: Rifdoor
[2023-02-25 22:45:50,448] INFO     processor/thread_8   Processed Malware Family: CobaltStrike
[2023-02-25 22:46:00,439] INFO     processor/thread_15  Processed Malware Family: GandCrab
[2023-02-25 22:46:00,952] INFO     processor/thread_6   Processed Malware Family: ContiRansomware
[2023-02-25 22:46:03,647] INFO     processor/thread_5   Processed Malware Family: Sodinokibi
[2023-02-25 22:46:23,466] INFO     processor/thread_13  Processed Malware Family: Salityv3
[2023-02-25 22:47:18,511] INFO     processor/thread_3   Processed Malware Family: njRAT
[2023-02-25 22:47:42,482] INFO     processor/thread_1   Processed Malware Family: Salityv4
[2023-02-25 22:47:42,855] INFO     processor/thread_8   Retrieved 302 bitcoin_address indicators from MISP.
[2023-02-25 22:47:42,861] INFO     processor/thread_9   Retrieved 7 coin_address indicators from MISP.
[2023-02-25 22:47:42,896] INFO     processor/thread_13  Retrieved 50 device_name indicators from MISP.
[2023-02-25 22:47:42,920] INFO     processor/thread_12  Retrieved 495 registry indicators from MISP.
[2023-02-25 22:47:42,922] INFO     processor/thread_9   Retrieved 29 service_name indicators from MISP.
[2023-02-25 22:47:43,086] INFO     processor/thread_3   Retrieved 1,740 hash_imphash indicators from MISP.
[2023-02-25 22:47:43,166] INFO     processor/thread_15  Retrieved 157 campaign_id indicators from MISP.
[2023-02-25 22:47:43,262] INFO     processor/thread_7   Retrieved 3,922 mutex_name indicators from MISP.
[2023-02-25 22:47:43,326] INFO     processor/thread_12  Retrieved 444 port indicators from MISP.
[2023-02-25 22:48:02,132] INFO     processor/thread_13  Retrieved 90,329 user_agent indicators from MISP.
[2023-02-25 22:48:09,459] INFO     processor/thread_4   Retrieved 199,762 file_name indicators from MISP.
[2023-02-25 22:48:12,832] INFO     processor/thread_5   Retrieved 199,762 file_path indicators from MISP.
[2023-02-25 22:48:15,722] INFO     processor/thread_10  Retrieved 18,980 email_address indicators from MISP.
[2023-02-25 22:48:57,764] INFO     processor/thread_2   Retrieved 507,424 hash_sha1 indicators from MISP.
[2023-02-25 22:49:12,653] INFO     processor/thread_11  Retrieved 501,155 email_subject indicators from MISP.
[2023-02-25 22:50:06,761] INFO     processor/thread_14  Retrieved 1,581,781 domain indicators from MISP.
[2023-02-25 22:52:16,148] INFO     processor/thread_1   Retrieved 1,864,786 hash_sha256 indicators from MISP.
[2023-02-25 22:53:12,248] INFO     processor/thread_0   Retrieved 2,489,070 hash_md5 indicators from MISP.
[2023-02-25 22:56:20,212] INFO     processor/thread_8   Retrieved 4,155,218 ip_address indicators from MISP.

<<<SNIP>>>

[2023-02-25 23:34:23,135] INFO     processor/thread_14  Updated Malware Family: GandCrab with 800 new indicators after 133.62 seconds.
[2023-02-25 23:34:30,640] INFO     processor/thread_3   Updated Indicator Type: SHA256 hashes with 822 new indicators after 178.74 seconds.
[2023-02-25 23:34:36,767] INFO     processor/thread_2   Updated Indicator Type: SHA1 hashes with 815 new indicators after 184.87 seconds.
[2023-02-25 23:34:36,767] DEBUG    processor/thread_2   Refreshing memory logged event: Indicator Type: SHA1 hashes
[2023-02-25 23:34:58,405] INFO     processor/thread_0   Updated Indicator Type: MD5 hashes with 814 new indicators after 206.53 seconds.
[2023-02-25 23:34:58,406] DEBUG    processor/thread_0   Refreshing memory logged event: Indicator Type: MD5 hashes
[2023-02-25 23:35:02,286] INFO     processor/thread_7   Updated Malware Family: njRAT with 229 new indicators after 210.18 seconds.
[2023-02-25 23:35:02,286] DEBUG    processor/thread_7   Refreshing memory logged event: Malware Family: njRAT
[2023-02-25 23:36:00,308] INFO     processor/thread_2   Indicator Type: SHA1 hashes refreshed in memory.
[2023-02-25 23:37:23,384] INFO     processor/thread_7   Malware Family: njRAT refreshed in memory.
[2023-02-25 23:39:09,909] INFO     processor/thread_0   Indicator Type: MD5 hashes refreshed in memory.
Traceback (most recent call last):
  File "misp_import.py", line 377, in <module>
    main()
  File "misp_import.py", line 356, in main
    importer.import_from_crowdstrike(int(settings["CrowdStrike"]["init_reports_days_before"]),
  File "/home/rx118r/src/crowdstrike/MISP-tools-main/cs_misp_import/importer.py", line 314, in import_from_crowdstrike
    self.indicators_importer.process_indicators(indicators_minutes_before)
  File "/home/rx118r/src/crowdstrike/MISP-tools-main/cs_misp_import/indicators.py", line 195, in process_indicators
    self.push_indicators(indicators_page)
  File "/home/rx118r/src/crowdstrike/MISP-tools-main/cs_misp_import/indicators.py", line 387, in push_indicators
    for cleaned in self.clean_laundry(len(batch), all_successes, f_failures, m_failures):
  File "/home/rx118r/src/crowdstrike/MISP-tools-main/cs_misp_import/indicators.py", line 363, in clean_laundry
    saved.append(fut.result())
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/rx118r/src/crowdstrike/MISP-tools-main/cs_misp_import/indicators.py", line 247, in event_thread
    self.misp.update_event(evt)
  File "/data/misp/venv/lib64/python3.8/site-packages/pymisp/api.py", line 417, in update_event
    r = self._prepare_request('POST', f'events/edit/{eid}' + ('/metadata:1' if metadata else ''), data=event)
  File "/data/misp/venv/lib64/python3.8/site-packages/pymisp/api.py", line 3705, in _prepare_request
    return self.__session.send(prepped, timeout=self.timeout, **settings)
  File "/data/misp/venv/lib64/python3.8/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/data/misp/venv/lib64/python3.8/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/data/misp/venv/lib64/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/data/misp/venv/lib64/python3.8/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/data/misp/venv/lib64/python3.8/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/http/client.py", line 1256, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/http/client.py", line 1302, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/http/client.py", line 1251, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/http/client.py", line 1050, in _send_output
    self.send(chunk)
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/http/client.py", line 972, in send
    self.sock.sendall(data)
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/ssl.py", line 1173, in send
    return self._sslobj.write(data)
OverflowError: string longer than 2147483647 bytes
jshcodes commented 1 year ago

Hi @packet-rat -

Does this happen consistently?

I'm attempting to recreate this, but my long running test hasn't run into it yet. (Fired it off last night hoping it was bad data that I needed to check for, but this is just a guess based upon where the failure occurred.)

packet-rat commented 1 year ago

I'll run it again...

Office: 732.615.5287 | Email: @.**@.>

From: Joshua Hiller @.> Sent: Monday, February 27, 2023 9:51 AM To: CrowdStrike/MISP-tools @.> Cc: MARONEY, PATRICK @.>; Mention @.> Subject: Re: [CrowdStrike/MISP-tools] OverflowError: string longer than 2147483647 bytes (Issue #102)

Hi @packet-rathttps://urldefense.com/v3/__https://github.com/packet-rat__;!!BhdT!nXdfcYFeG24ehxjNrmrwd9y3-_6avc-1OCwhHDFT_ppRi1-VZpbX4iLfwArnSpfR6Ka-qQleYwN5LCjPLHH8rOO2_wo$ -

Does this happen consistently?

I'm attempting to recreate this, but my long running test hasn't run into it yet. (Fired it off last night hoping it was bad data that I needed to check for, but this is just a guess based upon where the failure occurred.)

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/CrowdStrike/MISP-tools/issues/102*issuecomment-1446469918__;Iw!!BhdT!nXdfcYFeG24ehxjNrmrwd9y3-_6avc-1OCwhHDFT_ppRi1-VZpbX4iLfwArnSpfR6Ka-qQleYwN5LCjPLHH8_INR7mo$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AAYSFYPE4JZA5GLD3ZNVBATWZS5OPANCNFSM6AAAAAAVIFUN5U__;!!BhdT!nXdfcYFeG24ehxjNrmrwd9y3-_6avc-1OCwhHDFT_ppRi1-VZpbX4iLfwArnSpfR6Ka-qQleYwN5LCjPLHH8shpuhwM$. You are receiving this because you were mentioned.Message ID: @.***>

packet-rat commented 1 year ago

Weird:

2023-02-27 16:18:41,743] CRITICAL config authentication Invalid API credentials provided

python3 misp_import.py --fullmonty -d -v -p -nb [2023-02-27 16:18:41,738] INFO misp_tools MISP Import for CrowdStrike Threat Intelligence v0.6.8 [2023-02-27 16:18:41,738] INFO config CHECK CONFIG [2023-02-27 16:18:41,739] DEBUG config client_id value redacted, check config file [2023-02-27 16:18:41,739] DEBUG config client_secret value redacted, check config file [2023-02-27 16:18:41,739] DEBUG config crowdstrike_url US1 [2023-02-27 16:18:41,740] DEBUG config api_request_max 5000 [2023-02-27 16:18:41,740] DEBUG config api_enable_ssl True [2023-02-27 16:18:41,740] DEBUG config reports_timestamp_filename lastReportsUpdate.dat [2023-02-27 16:18:41,740] DEBUG config indicators_timestamp_filename lastIndicatorsUpdate.dat [2023-02-27 16:18:41,740] DEBUG config actors_timestamp_filename lastActorsUpdate.dat [2023-02-27 16:18:41,740] DEBUG config init_reports_days_before 365 [2023-02-27 16:18:41,740] DEBUG config init_indicators_minutes_before 20220 [2023-02-27 16:18:41,740] DEBUG config init_actors_days_before 365 [2023-02-27 16:18:41,740] DEBUG config reports_unique_tag CrowdStrike: REPORT [2023-02-27 16:18:41,740] DEBUG config indicators_unique_tag CrowdStrike: INDICATOR [2023-02-27 16:18:41,740] DEBUG config actors_unique_tag CrowdStrike: ADVERSARY [2023-02-27 16:18:41,740] DEBUG config reports_tags att:source="Crowdstrike.Report" [2023-02-27 16:18:41,740] DEBUG config indicators_tags att:source="Crowdstrike.Indicators" [2023-02-27 16:18:41,740] DEBUG config actors_tags att:source="Crowdstrike.Actors" [2023-02-27 16:18:41,740] DEBUG config unknown_mapping CrowdStrike:indicator:galaxy: UNATTRIBUTED [2023-02-27 16:18:41,740] DEBUG config unattributed_title Unattributed indicators: [2023-02-27 16:18:41,740] DEBUG config indicator_type_title Indicator Type: [2023-02-27 16:18:41,740] DEBUG config malware_family_title Malware Family: [2023-02-27 16:18:41,740] DEBUG config log_duplicates_as_sightings True [2023-02-27 16:18:41,740] DEBUG config misp_url https://3samisp [2023-02-27 16:18:41,740] DEBUG config misp_auth_key value redacted, check config file [2023-02-27 16:18:41,740] DEBUG config crowdstrike_org_uuid ca4f4b5d-db04-4a5e-a6de-e60636dc01be [2023-02-27 16:18:41,740] DEBUG config misp_enable_ssl False [2023-02-27 16:18:41,740] WARNING config misp_enable_ssl SSL is disabled for MISP API requests [2023-02-27 16:18:41,740] DEBUG config ind_attribute_batch_size 2500 [2023-02-27 16:18:41,740] DEBUG config event_save_memory_refresh_interval 180 [2023-02-27 16:18:41,740] DEBUG config max_threads 16 [2023-02-27 16:18:41,740] DEBUG config miss_track_file no_galaxy_mapping.log [2023-02-27 16:18:41,740] DEBUG config galaxies_map_file galaxy.ini [2023-02-27 16:18:41,740] DEBUG config tag_unknown_galaxy_maps True [2023-02-27 16:18:41,740] DEBUG config taxonomic_kill-chain True [2023-02-27 16:18:41,741] DEBUG config taxonomic_information-security-data-source True [2023-02-27 16:18:41,741] DEBUG config taxonomic_type True [2023-02-27 16:18:41,741] DEBUG config taxonomic_iep False [2023-02-27 16:18:41,741] DEBUG config taxonomic_iep2 True [2023-02-27 16:18:41,741] DEBUG config taxonomic_iep2_version False [2023-02-27 16:18:41,741] DEBUG config taxonomic_tlp True [2023-02-27 16:18:41,741] DEBUG config taxonomicworkflow True [2023-02-27 16:18:41,743] CRITICAL config authentication Invalid API credentials provided [2023-02-27 16:18:41,743] INFO config 1 configuration error found (1 warning) [2023-02-27 16:18:41,743] ERROR config [2023-02-27 16:18:41,743] ERROR config ____ ____ _ __ [2023-02-27 16:18:41,743] ERROR config | || | | |/ [ | || | | |__ | \ [2023-02-27 16:18:41,743] ERROR config | | | | | | _ ] | | | | | |_ |/ [2023-02-27 16:18:41,743] ERROR config [2023-02-27 16:18:41,743] INFO misp_tools FINISHED Invalid configuration specified, unable to continue.

Office: 732.615.5287 | Email: @.**@.>

From: MARONEY, PATRICK @.> Sent: Monday, February 27, 2023 11:17 AM To: CrowdStrike/MISP-tools @.>; CrowdStrike/MISP-tools @.> Cc: Mention @.> Subject: Re: [CrowdStrike/MISP-tools] OverflowError: string longer than 2147483647 bytes (Issue #102)

I'll run it again...

Patrick Maroney Principal - Cybersecurity Chief Security Office AT&T Services, Inc. 200 S Laurel Ave, Middletown, NJ 07748

Office: 732.615.5287 | Email: @.**@.>

From: Joshua Hiller @.> Sent: Monday, February 27, 2023 9:51 AM To: CrowdStrike/MISP-tools @.> Cc: MARONEY, PATRICK @.>; Mention @.> Subject: Re: [CrowdStrike/MISP-tools] OverflowError: string longer than 2147483647 bytes (Issue #102)

Hi @packet-rathttps://urldefense.com/v3/__https://github.com/packet-rat__;!!BhdT!nXdfcYFeG24ehxjNrmrwd9y3-_6avc-1OCwhHDFT_ppRi1-VZpbX4iLfwArnSpfR6Ka-qQleYwN5LCjPLHH8rOO2_wo$ -

Does this happen consistently?

I'm attempting to recreate this, but my long running test hasn't run into it yet. (Fired it off last night hoping it was bad data that I needed to check for, but this is just a guess based upon where the failure occurred.)

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/CrowdStrike/MISP-tools/issues/102*issuecomment-1446469918__;Iw!!BhdT!nXdfcYFeG24ehxjNrmrwd9y3-_6avc-1OCwhHDFT_ppRi1-VZpbX4iLfwArnSpfR6Ka-qQleYwN5LCjPLHH8_INR7mo$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AAYSFYPE4JZA5GLD3ZNVBATWZS5OPANCNFSM6AAAAAAVIFUN5U__;!!BhdT!nXdfcYFeG24ehxjNrmrwd9y3-_6avc-1OCwhHDFT_ppRi1-VZpbX4iLfwArnSpfR6Ka-qQleYwN5LCjPLHH8shpuhwM$. You are receiving this because you were mentioned.Message ID: @.***>

jshcodes commented 1 year ago

It looks like it can't see your configuration file.

packet-rat commented 1 year ago

re: "Invalid API credentials provided"

The lab server cannot reach your API. A better connectivity test to discriminate between connection vs. credentialing could prevent mild-goose-chases.

Note that the original issue remains (presumably since we can't retest until out lab access is fixed.)

OverflowError: string longer than 2147483647 bytes

packet-rat commented 1 year ago

Deleted .dat files:

Ran just indictors:

python3 misp_import.py -d -p -v -i

(venv) [rx118r@md2nj01di:~/src/crowdstrike/MISP-tools-main]$ python3 misp_import.py -d -p -v -r
[2023-03-06 16:07:18,046] INFO     misp_tools    
[2023-03-06 16:07:18,046] INFO     misp_tools    '##::::'##:'####::'######::'########:::::'########::'#######:::'#######::'##::::::::'######::
[2023-03-06 16:07:18,047] INFO     misp_tools     ###::'###:. ##::'##... ##: ##.... ##::::... ##..::'##.... ##:'##.... ##: ##:::::::'##... ##:
[2023-03-06 16:07:18,047] INFO     misp_tools     ####'####:: ##:: ##:::..:: ##:::: ##::::::: ##:::: ##:::: ##: ##:::: ##: ##::::::: ##:::..::
[2023-03-06 16:07:18,047] INFO     misp_tools     ## ### ##:: ##::. ######:: ########:::::::: ##:::: ##:::: ##: ##:::: ##: ##:::::::. ######::
[2023-03-06 16:07:18,047] INFO     misp_tools     ##. #: ##:: ##:::..... ##: ##.....::::::::: ##:::: ##:::: ##: ##:::: ##: ##::::::::..... ##:
[2023-03-06 16:07:18,047] INFO     misp_tools     ##:.:: ##:: ##::'##::: ##: ##:::::::::::::: ##:::: ##:::: ##: ##:::: ##: ##:::::::'##::: ##:
[2023-03-06 16:07:18,047] INFO     misp_tools     ##:::: ##:'####:. ######:: ##:::::::::::::: ##::::. #######::. #######:: ########:. ######::
[2023-03-06 16:07:18,047] INFO     misp_tools    ..:::::..::....:::......:::..:::::::::::::::..::::::.......::::.......:::........:::......:::
[2023-03-06 16:07:18,047] INFO     misp_tools               _____
[2023-03-06 16:07:18,047] INFO     misp_tools                /  '
[2023-03-06 16:07:18,047] INFO     misp_tools             ,-/-,__ __
[2023-03-06 16:07:18,047] INFO     misp_tools            (_/  (_)/ (_
[2023-03-06 16:07:18,047] INFO     misp_tools                         _______                        __ _______ __        __ __
[2023-03-06 16:07:18,047] INFO     misp_tools                        |   _   .----.-----.--.--.--.--|  |   _   |  |_.----|__|  |--.-----.
[2023-03-06 16:07:18,047] INFO     misp_tools                        |.  1___|   _|  _  |  |  |  |  _  |   1___|   _|   _|  |    <|  -__|
[2023-03-06 16:07:18,047] INFO     misp_tools                        |.  |___|__| |_____|________|_____|____   |____|__| |__|__|__|_____|
[2023-03-06 16:07:18,047] INFO     misp_tools                        |:  1   |                         |:  1   |
[2023-03-06 16:07:18,047] INFO     misp_tools                        |::.. . |                         |::.. . |  Threat Intelligence v0.6.8
[2023-03-06 16:07:18,047] INFO     misp_tools                        `-------'                         `-------'
[2023-03-06 16:07:18,047] INFO     misp_tools    
[2023-03-06 16:07:18,047] INFO     config  
[2023-03-06 16:07:18,047] INFO     config  _______ _     _ _______ _______ _     _      _______  _____  __   _ _______ _____  ______
[2023-03-06 16:07:18,047] INFO     config  |       |_____| |______ |       |____/       |       |     | | \  | |______   |   |  ____
[2023-03-06 16:07:18,047] INFO     config  |_____  |     | |______ |_____  |    \_      |_____  |_____| |  \_| |       __|__ |_____|
[2023-03-06 16:07:18,047] INFO     config  
[2023-03-06 16:07:18,048] DEBUG    config  client_id                                   value redacted, check config file
[2023-03-06 16:07:18,048] DEBUG    config  client_secret                               value redacted, check config file
[2023-03-06 16:07:18,048] DEBUG    config  crowdstrike_url                             US1
[2023-03-06 16:07:18,048] DEBUG    config  api_request_max                             5000
[2023-03-06 16:07:18,048] DEBUG    config  api_enable_ssl                              True
[2023-03-06 16:07:18,048] DEBUG    config  reports_timestamp_filename                  lastReportsUpdate.dat
[2023-03-06 16:07:18,048] DEBUG    config  indicators_timestamp_filename               lastIndicatorsUpdate.dat
[2023-03-06 16:07:18,048] DEBUG    config  actors_timestamp_filename                   lastActorsUpdate.dat
[2023-03-06 16:07:18,048] DEBUG    config  init_reports_days_before                    365
[2023-03-06 16:07:18,048] DEBUG    config  init_indicators_minutes_before              20220
[2023-03-06 16:07:18,048] DEBUG    config  init_actors_days_before                     365
[2023-03-06 16:07:18,048] DEBUG    config  reports_unique_tag                          CrowdStrike: REPORT
[2023-03-06 16:07:18,048] DEBUG    config  indicators_unique_tag                       CrowdStrike: INDICATOR
[2023-03-06 16:07:18,048] DEBUG    config  actors_unique_tag                           CrowdStrike: ADVERSARY
[2023-03-06 16:07:18,048] DEBUG    config  reports_tags                                att:source="Crowdstrike.Report"
[2023-03-06 16:07:18,048] DEBUG    config  indicators_tags                             att:source="Crowdstrike.Indicators"
[2023-03-06 16:07:18,048] DEBUG    config  actors_tags                                 att:source="Crowdstrike.Actors"
[2023-03-06 16:07:18,048] DEBUG    config  unknown_mapping                             CrowdStrike:indicator:galaxy: UNATTRIBUTED
[2023-03-06 16:07:18,048] DEBUG    config  unattributed_title                          Unattributed indicators:
[2023-03-06 16:07:18,048] DEBUG    config  indicator_type_title                        Indicator Type:
[2023-03-06 16:07:18,048] DEBUG    config  malware_family_title                        Malware Family:
[2023-03-06 16:07:18,048] DEBUG    config  log_duplicates_as_sightings                 True
[2023-03-06 16:07:18,048] DEBUG    config  misp_url                                    https://3samisp
[2023-03-06 16:07:18,048] DEBUG    config  misp_auth_key                               value redacted, check config file
[2023-03-06 16:07:18,048] DEBUG    config  crowdstrike_org_uuid                        ca4f4b5d-db04-4a5e-a6de-e60636dc01be
[2023-03-06 16:07:18,048] DEBUG    config  misp_enable_ssl                             False
[2023-03-06 16:07:18,048] WARNING  config  misp_enable_ssl                             SSL is disabled for MISP API requests
[2023-03-06 16:07:18,048] DEBUG    config  ind_attribute_batch_size                    2500
[2023-03-06 16:07:18,048] DEBUG    config  event_save_memory_refresh_interval          180
[2023-03-06 16:07:18,048] DEBUG    config  max_threads                                 16
[2023-03-06 16:07:18,049] DEBUG    config  miss_track_file                             no_galaxy_mapping.log
[2023-03-06 16:07:18,049] DEBUG    config  galaxies_map_file                           galaxy.ini
[2023-03-06 16:07:18,049] DEBUG    config  tag_unknown_galaxy_maps                     True
[2023-03-06 16:07:18,049] DEBUG    config  taxonomic_kill-chain                        True
[2023-03-06 16:07:18,049] DEBUG    config  taxonomic_information-security-data-source  True
[2023-03-06 16:07:18,049] DEBUG    config  taxonomic_type                              True
[2023-03-06 16:07:18,049] DEBUG    config  taxonomic_iep                               False
[2023-03-06 16:07:18,049] DEBUG    config  taxonomic_iep2                              True
[2023-03-06 16:07:18,049] DEBUG    config  taxonomic_iep2_version                      False
[2023-03-06 16:07:18,049] DEBUG    config  taxonomic_tlp                               True
[2023-03-06 16:07:18,049] DEBUG    config  taxonomic_workflow                          True
[2023-03-06 16:07:18,556] INFO     config  No configuration errors found (1 warning)
[2023-03-06 16:07:18,556] INFO     config  
[2023-03-06 16:07:18,556] INFO     config  ____ _  _ ____ ____ _  _ ____    ___  ____ ____ ____ ____ ___
[2023-03-06 16:07:18,556] INFO     config  |    |__| |___ |    |_/  [__     |__] |__| [__  [__  |___ |  \
[2023-03-06 16:07:18,556] INFO     config  |___ |  | |___ |___ | \_ ___]    |    |  | ___] ___] |___ |__/
[2023-03-06 16:07:18,556] INFO     config  
[2023-03-06 16:07:20,105] INFO     processor/main       
[2023-03-06 16:07:20,105] INFO     processor/main       _____ _______  _____   _____   ______ _______
[2023-03-06 16:07:20,106] INFO     processor/main         |   |  |  | |_____] |     | |_____/    |
[2023-03-06 16:07:20,106] INFO     processor/main       __|__ |  |  | |       |_____| |    \_    |
[2023-03-06 16:07:20,106] INFO     processor/main       
[2023-03-06 16:07:20,106] INFO     processor/main       
[2023-03-06 16:07:20,106] INFO     processor/main        ____     ___  ____    ___   ____  ______  _____
[2023-03-06 16:07:20,106] INFO     processor/main       |    \   /  _]|    \  /   \ |    \|      T/ ___/
[2023-03-06 16:07:20,106] INFO     processor/main       |  D  ) /  [_ |  o  )Y     Y|  D  )      (   \_
[2023-03-06 16:07:20,106] INFO     processor/main       |    / Y    _]|   _/ |  O  ||    /l_j  l_j\__  T
[2023-03-06 16:07:20,106] INFO     processor/main       |    \ |   [_ |  |   |     ||    \  |  |  /  \ |
[2023-03-06 16:07:20,106] INFO     processor/main       |  .  Y|     T|  |   l     !|  .  Y |  |  \    |
[2023-03-06 16:07:20,106] INFO     processor/main       l__j\_jl_____jl__j    \___/ l__j\_j l__j   \___j
[2023-03-06 16:07:20,106] INFO     processor/main       
[2023-03-06 16:07:20,107] INFO     processor/main       Starting import of CrowdStrike Threat Intelligence reports as events (past 365 days).
[2023-03-06 16:07:20,107] INFO     processor/main       Retrieving all available report types.
[2023-03-06 16:07:24,130] INFO     processor/main       Retrieved 45 total reports from the Crowdstrike Intel API.
[2023-03-06 16:07:24,130] INFO     processor/main       Found 2340 pre-existing CrowdStrike reports within the MISP instance.
[2023-03-06 16:07:29,129] INFO     processor/main       Retrieved extended report details for 39 reports.
[2023-03-06 16:07:32,486] INFO     processor/main       371 related indicators found.
[2023-03-06 16:07:32,489] DEBUG    processor/thread_1   Retrieved 11 indicators detailed within report CSA-230312
[2023-03-06 16:07:32,527] DEBUG    processor/thread_8   Retrieved 7 indicators detailed within report CSA-230293
[2023-03-06 16:07:32,565] DEBUG    processor/thread_12  Retrieved 30 indicators detailed within report CSA-230314
[2023-03-06 16:07:32,600] DEBUG    processor/thread_14  Retrieved 33 indicators detailed within report CSA-230297
[2023-03-06 16:07:32,812] DEBUG    processor/thread_5   CSIT-23059 Emerging Trends in Uzbekistan Hacktivism report created.
[2023-03-06 16:07:32,813] DEBUG    processor/thread_5   Retrieved 42 indicators detailed within report CSA-230328
[2023-03-06 16:07:32,816] DEBUG    processor/thread_4   CSIT-23070 Operational Profile of Anti-Iranian Government Hacktivist Group Black Reward report created.
[2023-03-06 16:07:32,849] DEBUG    processor/thread_6   CSIT-23072 ATM Attacks Fluctuate and Resurge After COVID-19 Lockdowns End report created.
[2023-03-06 16:07:32,858] DEBUG    processor/thread_11  CSA-230323 Oracle Web Logic Vulnerability (CVE-2023-21839) Exploit Proof-of-Concept Released, Automated Exploitation Attempts Likely report created.
[2023-03-06 16:07:32,861] DEBUG    processor/thread_13  CSA-230325 CCP Releases “Global Security Initiative” Paper Outlining Chinese Alternative to U.S.-Led Global Security Architecture report created.
[2023-03-06 16:07:32,864] DEBUG    processor/thread_9   CSA-230321 Iran Expands Military Electronics Proliferation, Broadens Alternative Technology Supply Chain report created.
[2023-03-06 16:07:32,869] DEBUG    processor/thread_1   CSA-230312 Watermeloader Rust Loader Protected with Modified Exocet Crypter; Distribution Overlap with CARBON SPIDER report created.
[2023-03-06 16:07:32,876] DEBUG    processor/thread_8   CSA-230293 InnateSpark Continues to Deliver AvantGarde in February 2023 After Brief Hiatus; Likely Targeting Apple Devices report created.
[2023-03-06 16:07:32,880] DEBUG    processor/thread_9   Retrieved 3 indicators detailed within report CSA-230322
[2023-03-06 16:07:32,980] DEBUG    processor/thread_3   Retrieved 15 indicators detailed within report CSA-230310
[2023-03-06 16:07:33,046] DEBUG    processor/thread_9   CSA-230322 Opportunistic eCrime Actor Exploits ManageEngine and KACE, Deploys ScreenConnect report created.
[2023-03-06 16:07:33,055] DEBUG    processor/thread_14  CSA-230297 China-Nexus Adversary Targets Telecommunication Entities with Reptile Rootkit and SideWalk Malware report created.
[2023-03-06 16:07:33,177] DEBUG    processor/thread_12  CSA-230314 Industry Reporting Details ForgedCombine Activity Targeting Telecommunications Entities in the Middle East and Likely Afghanistan report created.
[2023-03-06 16:07:33,178] DEBUG    processor/thread_12  Retrieved 9 indicators detailed within report CSA-230329
[2023-03-06 16:07:33,195] DEBUG    processor/thread_14  CSA-230335 Iran Emphasizes Online Media Messaging Control to Strengthen Cognitive Warfare report created.
[2023-03-06 16:07:33,195] DEBUG    processor/thread_14  Retrieved 15 indicators detailed within report CSA-230317
[2023-03-06 16:07:33,215] DEBUG    processor/thread_5   CSA-230328 Shindig Updates its Loader report created.
[2023-03-06 16:07:33,284] DEBUG    processor/thread_3   CSA-230310 RECESS SPIDER Leverages Compromised VPN Credentials for Initial Access report created.
[2023-03-06 16:07:33,401] DEBUG    processor/thread_6   Retrieved 10 indicators detailed within report CSA-230316
[2023-03-06 16:07:33,465] DEBUG    processor/thread_12  CSA-230329 Fsociety Updates Tooling, Including New Obfuscation Process to Hide Malicious Infrastructure report created.
[2023-03-06 16:07:33,500] DEBUG    processor/thread_14  CSA-230317 Mallox Ransomware Activity Identified; the Group Recently Began Recruiting Pentesters via an Underground Forum report created.
[2023-03-06 16:07:33,591] DEBUG    processor/thread_12  CSA-230331 Chinese and Russian Propagandists Exploit Ohio Train Derailment report created.

SNIP

[2023-03-06 21:33:50,399] INFO     processor/thread_3   Updated Indicator Type: SHA256 hashes with 824 new indicators after 151.64 seconds.
[2023-03-06 21:33:56,574] INFO     processor/thread_2   Updated Indicator Type: SHA1 hashes with 810 new indicators after 157.84 seconds.
[2023-03-06 21:34:06,270] INFO     processor/thread_0   Updated Indicator Type: MD5 hashes with 811 new indicators after 167.55 seconds.
[2023-03-06 21:34:36,704] INFO     processor/thread_15  Updated Malware Family: Sodinokibi with 337 new indicators after 176.87 seconds.
Traceback (most recent call last):
  File "misp_import.py", line 377, in <module>
    main()
  File "misp_import.py", line 356, in main
    importer.import_from_crowdstrike(int(settings["CrowdStrike"]["init_reports_days_before"]),
  File "/home/rx118r/src/crowdstrike/MISP-tools-main/cs_misp_import/importer.py", line 314, in import_from_crowdstrike
    self.indicators_importer.process_indicators(indicators_minutes_before)
  File "/home/rx118r/src/crowdstrike/MISP-tools-main/cs_misp_import/indicators.py", line 195, in process_indicators
    self.push_indicators(indicators_page)
  File "/home/rx118r/src/crowdstrike/MISP-tools-main/cs_misp_import/indicators.py", line 387, in push_indicators
    for cleaned in self.clean_laundry(len(batch), all_successes, f_failures, m_failures):
  File "/home/rx118r/src/crowdstrike/MISP-tools-main/cs_misp_import/indicators.py", line 363, in clean_laundry
    saved.append(fut.result())
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/rx118r/src/crowdstrike/MISP-tools-main/cs_misp_import/indicators.py", line 247, in event_thread
    self.misp.update_event(evt)
  File "/data/misp/venv/lib64/python3.8/site-packages/pymisp/api.py", line 417, in update_event
    r = self._prepare_request('POST', f'events/edit/{eid}' + ('/metadata:1' if metadata else ''), data=event)
  File "/data/misp/venv/lib64/python3.8/site-packages/pymisp/api.py", line 3705, in _prepare_request
    return self.__session.send(prepped, timeout=self.timeout, **settings)
  File "/data/misp/venv/lib64/python3.8/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/data/misp/venv/lib64/python3.8/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/data/misp/venv/lib64/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/data/misp/venv/lib64/python3.8/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/data/misp/venv/lib64/python3.8/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/http/client.py", line 1256, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/http/client.py", line 1302, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/http/client.py", line 1251, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/http/client.py", line 1050, in _send_output
    self.send(chunk)
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/http/client.py", line 972, in send
    self.sock.sendall(data)
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/ssl.py", line 1173, in send
    return self._sslobj.write(data)
OverflowError: string longer than 2147483647 bytes
(venv) [rx118r@md2nj01di:~/src/crowdstrike/MISP-tools-main]$
packet-rat commented 1 year ago

I see quite a few hits on that error (relating to trying to load a chunk-o-stuff over 2GBs).

For example:

https://www.dropboxforum.com/t5/Dropbox-API-Support-Feedback/python-upload-file-OverflowError/td-p/437475

packet-rat commented 1 year ago

Do you want me to do a complete wipe and try from scratch again?

Change my settings?

init_reports_days_before = 365 init_indicators_minutes_before = 20220 init_actors_days_before = 365

jshcodes commented 1 year ago

Try dropping init_indicators_minutes_before down to 300.

If that doesn't get you past the error, then try clearing. (I also am finding hits related to overall update size. We may need to consider chunking indicator events that exceed a certain attribute count.)

packet-rat commented 1 year ago

Please note that the indicator runs are typically taking in excess of 5 hours to complete/fail.

Out of curiosity: why do you appear to pull all of the Attributes from all sources?


[2023-03-07 16:11:46,497] INFO     processor/thread_9   Retrieved 7 coin_address indicators from MISP.
[2023-03-07 16:11:46,651] INFO     processor/thread_3   Retrieved 1,740 hash_imphash indicators from MISP.
[2023-03-07 16:11:46,658] INFO     processor/thread_13  Retrieved 50 device_name indicators from MISP.
[2023-03-07 16:11:46,894] INFO     processor/thread_7   Retrieved 3,922 mutex_name indicators from MISP.
[2023-03-07 16:11:46,908] INFO     processor/thread_8   Retrieved 302 bitcoin_address indicators from MISP.
[2023-03-07 16:11:47,264] INFO     processor/thread_12  Retrieved 495 registry indicators from MISP.
[2023-03-07 16:11:48,801] INFO     processor/thread_9   Retrieved 171 campaign_id indicators from MISP.
[2023-03-07 16:11:48,816] INFO     processor/thread_3   Retrieved 29 service_name indicators from MISP.
[2023-03-07 16:11:49,486] INFO     processor/thread_7   Retrieved 446 port indicators from MISP.
[2023-03-07 16:12:09,651] INFO     processor/thread_13  Retrieved 90,329 user_agent indicators from MISP.
[2023-03-07 16:12:23,738] INFO     processor/thread_4   Retrieved 199,917 file_name indicators from MISP.
[2023-03-07 16:12:27,594] INFO     processor/thread_5   Retrieved 199,917 file_path indicators from MISP.
[2023-03-07 16:12:31,243] INFO     processor/thread_10  Retrieved 21,408 email_address indicators from MISP.
[2023-03-07 16:12:49,347] INFO     processor/thread_2   Retrieved 512,139 hash_sha1 indicators from MISP.
[2023-03-07 16:13:39,268] INFO     processor/thread_11  Retrieved 547,578 email_subject indicators from MISP.
[2023-03-07 16:14:19,051] INFO     processor/thread_14  Retrieved 1,588,537 domain indicators from MISP.
[2023-03-07 16:16:44,599] INFO     processor/thread_1   Retrieved 2,112,980 hash_sha256 indicators from MISP.
[2023-03-07 16:17:50,401] INFO     processor/thread_0   Retrieved 2,737,593 hash_md5 indicators from MISP.
[2023-03-07 16:20:39,068] INFO     processor/thread_15  Retrieved 4,214,016 ip_address indicators from MISP.
[2023-03-07 16:42:20,474] INFO     processor/thread_6   Retrieved 14,758,445 url indicators from MISP.
[2023-03-07 16:42:35,343] INFO     processor/main       Found 0 pre-existing indicators within CrowdStrike reports.
[2023-03-07 16:42:48,466] INFO     processor/main       Starting import of CrowdStrike indicators into MISP.
[2023-03-07 16:43:07,859] INFO     processor/main       Retrieved 5,000 of 38,205 remaining indicators.
[2023-03-07 16:43:07,859] DEBUG    processor/main       Configuration states we should process batches of 2,500 indicators.
jshcodes commented 1 year ago

Out of curiosity: why do you appear to pull all of the Attributes from all sources?

Dupe checking. This has changed over the past few versions, at some point this will get revisited. (Reports and Adversaries will probably still populate this way.)

packet-rat commented 1 year ago

So what do you do (or not do) if one of your competitors/alternate sources have stated than an IOC is bad?