Converting YAML to Json

hawki999 commented 3 years ago

Greetings I get the following error when running the conversion any ideas please

[!] The below YAML file contains possible errors. It's recommended to check via the '--health' argument or using the option in the interactive menu:

/mnt/c/Users/craig/Downloads/data-sources-new.yaml Traceback (most recent call last): File "dettect.py", line 365, in _menu(_init_menu()) File "dettect.py", line 254, in _menu generate_data_sources_layer(file_ds, args.output_filename, args.layer_name, args.platform) File "/home/craig/DeTTECT/data_source_mapping.py", line 24, in generate_data_sources_layer my_techniques = _map_and_colorize_techniques(my_data_sources, platform, exceptions) File "/home/craig/DeTTECT/data_source_mapping.py", line 261, in _map_and_colorize_techniques total_ds_count = _count_applicable_data_sources(t, applicable_data_sources) File "/home/craig/DeTTECT/data_source_mapping.py", line 240, in _count_applicable_data_sources ds = ds.split(':')[1][1:] IndexError: list index out of range

marcusbakker commented 3 years ago

@hawki999 could you please have a look at this comment from @rubinatorz. As your issues seem to be the same.

hawki999 commented 3 years ago

Afternoon Thanks Marcus

Unfortunately i still get the same error after the proposed workaround

[!] Data source: 'Logon Session Creation' is MISSING from the YAML file [!] Data source: 'Scheduled Job Modification' is MISSING from the YAML file [!] Data source: 'Instance Stop' is MISSING from the YAML file [!] Data source: 'Cloud Service Modification' is MISSING from the YAML file [!] Data source: 'File Modification' is MISSING from the YAML file [!] Data source: 'Logon Session Metadata' is MISSING from the YAML file [!] Data source: 'Firewall Metadata' is MISSING from the YAML file [!] Data source: 'Volume Modification' is MISSING from the YAML file [!] Data source: 'Web Credential Usage' is MISSING from the YAML file [!] Data source: 'Pod Enumeration' is MISSING from the YAML file [!] Data source: 'File Deletion' is MISSING from the YAML file [!] Data source: 'Container Start' is MISSING from the YAML file [!] Data source: 'Cloud Service Metadata' is MISSING from the YAML file [!] Data source: 'Container Creation' is MISSING from the YAML file [!] Data source: 'User Account Modification' is MISSING from the YAML file [!] Data source: 'Volume Metadata' is MISSING from the YAML file [!] Data source: 'Active Directory Object Access' is MISSING from the YAML file [!] Data source: 'Host Status' is MISSING from the YAML file [!] Data source: 'Command Execution' is MISSING from the YAML file [!] Data source: 'Pod Creation' is MISSING from the YAML file [!] Data source: 'OS API Execution' is MISSING from the YAML file [!] Data source: 'Image Deletion' is MISSING from the YAML file [!] Data source: 'Driver Metadata' is MISSING from the YAML file [!] Data source: 'Instance Enumeration' is MISSING from the YAML file [!] Data source: 'Pod Modification' is MISSING from the YAML file [!] Data source: 'Network Traffic Flow' is MISSING from the YAML file [!] Data source: 'Active Directory Object Modification' is MISSING from the YAML file [!] Data source: 'Process Metadata' is MISSING from the YAML file [!] Data source: 'Firmware Modification' is MISSING from the YAML file [!] Data source: 'User Account Metadata' is MISSING from the YAML file [!] Data source: 'Snapshot Deletion' is MISSING from the YAML file [!] Data source: 'Scheduled Job Creation' is MISSING from the YAML file [!] Data source: 'Windows Registry Key Creation' is MISSING from the YAML file [!] Data source: 'User Account Deletion' is MISSING from the YAML file [!] Data source: 'File Creation' is MISSING from the YAML file [!] Data source: 'Process Termination' is MISSING from the YAML file [!] Data source: 'Instance Modification' is MISSING from the YAML file [!] Data source: 'Service Modification' is MISSING from the YAML file [!] Data source: 'Kernel Module Load' is MISSING from the YAML file [!] Data source: 'User Account Authentication' is MISSING from the YAML file [!] Data source: 'Container Enumeration' is MISSING from the YAML file [!] Data source: 'File Content' is MISSING from the YAML file [!] Data source: 'Image Modification' is MISSING from the YAML file [!] Data source: 'Network Traffic Content' is MISSING from the YAML file [!] Data source: 'Instance Metadata' is MISSING from the YAML file [!] Data source: 'Process Creation' is MISSING from the YAML file [!] Data source: 'Cloud Storage Access' is MISSING from the YAML file [!] Data source: 'Module Load' is MISSING from the YAML file [!] Data source: 'Web Credential Creation' is MISSING from the YAML file [!] Data source: 'Pod Metadata' is MISSING from the YAML file [!] Data source: 'File Metadata' is MISSING from the YAML file [!] Data source: 'Instance Start' is MISSING from the YAML file [!] Data source: 'Cluster Metadata' is MISSING from the YAML file [!] Data source: 'Cloud Storage Metadata' is MISSING from the YAML file [!] Data source: 'Windows Registry Key Deletion' is MISSING from the YAML file [!] Data source: 'Windows Registry Key Modification' is MISSING from the YAML file [!] Data source: 'Instance Deletion' is MISSING from the YAML file [!] Data source: 'Active Directory Object Deletion' is MISSING from the YAML file [!] Data source: 'Process Access' is MISSING from the YAML file [!] Data source: 'Active Directory Credential Request' is MISSING from the YAML file [!] Data source: 'Script Execution' is MISSING from the YAML file [!] Data source: 'Snapshot Enumeration' is MISSING from the YAML file [!] Data source: 'Cloud Storage Creation' is MISSING from the YAML file [!] Data source: 'Windows Registry Key Access' is MISSING from the YAML file [!] Data source: 'Service Metadata' is MISSING from the YAML file [!] Data source: 'Service Creation' is MISSING from the YAML file [!] Data source: 'Cloud Storage Modification' is MISSING from the YAML file [!] Data source: 'Volume Enumeration' is MISSING from the YAML file [!] Data source: 'Snapshot Metadata' is MISSING from the YAML file [!] Data source: 'Volume Deletion' is MISSING from the YAML file [!] Data source: 'WMI Creation' is MISSING from the YAML file [!] Data source: 'Firewall Rule Modification' is MISSING from the YAML file [!] Data source: 'Application Log Content' is MISSING from the YAML file [!] Data source: 'Snapshot Creation' is MISSING from the YAML file [!] Data source: 'Driver Load' is MISSING from the YAML file [!] Data source: 'Active Directory Object Creation' is MISSING from the YAML file [!] Data source: 'Network Connection Creation' is MISSING from the YAML file [!] Data source: 'Drive Modification' is MISSING from the YAML file [!] Data source: 'Volume Creation' is MISSING from the YAML file [!] Data source: 'Firewall Enumeration' is MISSING from the YAML file [!] Data source: 'Group Modification' is MISSING from the YAML file [!] Data source: 'Network Share Access' is MISSING from the YAML file [!] Data source: 'Group Metadata' is MISSING from the YAML file [!] Data source: 'Container Metadata' is MISSING from the YAML file [!] Data source: 'User Account Creation' is MISSING from the YAML file [!] Data source: 'Snapshot Modification' is MISSING from the YAML file [!] Data source: 'Drive Access' is MISSING from the YAML file [!] Data source: 'Drive Creation' is MISSING from the YAML file [!] Data source: 'File Access' is MISSING from the YAML file [!] Data source: 'Scheduled Job Metadata' is MISSING from the YAML file [!] Data source: 'Cloud Storage Enumeration' is MISSING from the YAML file [!] Data source: 'Image Metadata' is MISSING from the YAML file [!] Data source: 'Image Creation' is MISSING from the YAML file [!] Data source: 'Cloud Service Disable' is MISSING from the YAML file [!] Data source: 'Cloud Service Enumeration' is MISSING from the YAML file [!] Data source: 'Instance Creation' is MISSING from the YAML file [!] Data source: 'Group Enumeration' is MISSING from the YAML file Traceback (most recent call last): File "dettect.py", line 365, in _menu(_init_menu()) File "dettect.py", line 254, in _menu generate_data_sources_layer(file_ds, args.output_filename, args.layer_name, args.platform) File "/home/craig/DeTTECT/data_source_mapping.py", line 24, in generate_data_sources_layer my_techniques = _map_and_colorize_techniques(my_data_sources, platform, exceptions) File "/home/craig/DeTTECT/data_source_mapping.py", line 261, in _map_and_colorize_techniques total_ds_count = _count_applicable_data_sources(t, applicable_data_sources) File "/home/craig/DeTTECT/data_source_mapping.py", line 240, in _count_applicable_data_sources ds = ds.split(':')[1][1:] IndexError: list index out of range

rubinatorz commented 3 years ago

hi @hawki999

How does your data source YAML file look like and what specifiek dettect.py command line are you using?

hawki999 commented 3 years ago

python3 dettect.py ds -fd /mnt/c/Users/craig/Downloads/data-sources-new.yaml -l --health

version: 1 file_type: data-source-administration name: example platform:

all data_sources:
data_source_name: Firewall Disable date_registered: null date_connected: null products: [] available_for_data_analytics: true comment: '' data_quality: device_completeness: 1 data_field_completeness: 1 timeliness: 1 consistency: 1 retention: 1
data_source_name: Cloud Storage Deletion date_registered: null date_connected: null products: [] available_for_data_analytics: true comment: '' data_quality: device_completeness: 1 data_field_completeness: 1 timeliness: 1 consistency: 1 retention: 1

rubinatorz commented 3 years ago

hi @hawki999

Thanks, I tried the exact same command with the exact same YAML file and get this:

python3 dettect.py ds -fd ds-error.yaml -l --health
[!] Data source: 'Cloud Storage Modification' is MISSING from the YAML file
...
[!] Data source: 'Network Traffic Content' is MISSING from the YAML file
File written:   output/data_sources_example.json

I removed a bunch of lines in this output to keep it short. So all working here...

I can't get my finger on it...

Can you run the "pip3 list" command and send me the output?

hawki999 commented 3 years ago

Hi Ruben

please find the list below - best regards

Package Version

antlr4-python3-runtime 4.8 attackcti 0.3.4.3 attrs 19.3.0 Automat 0.8.0 blinker 1.4 certifi 2019.11.28 chardet 3.0.4 Click 7.0 cloud-init 21.2 colorama 0.4.3 command-not-found 0.3 configobj 5.0.6 constantly 15.1.0 cryptography 2.8 dbus-python 1.2.16 distro 1.4.0 distro-info 0.23ubuntu1 entrypoints 0.3 eql 0.9.9 httplib2 0.14.0 hyperlink 19.0.0 idna 2.8 importlib-metadata 1.5.0 incremental 16.10.1 Jinja2 2.10.1 jsonpatch 1.22 jsonpointer 2.0 jsonschema 3.2.0 keyring 18.0.1 language-selector 0.1 lark-parser 0.11.3 launchpadlib 1.10.13 lazr.restfulclient 0.14.2 lazr.uri 1.0.3 MarkupSafe 1.1.0 more-itertools 4.2.0 netifaces 0.10.4 numpy 1.20.3 oauthlib 3.1.0 pandas 1.2.4 pexpect 4.6.0 pip 20.0.2 plotly 5.0.0 pyasn1 0.4.2 pyasn1-modules 0.2.1 PyGObject 3.36.0 PyHamcrest 1.9.0 PyJWT 1.7.1 pymacaroons 0.13.0 PyNaCl 1.3.0 pyOpenSSL 19.0.0 pyrsistent 0.15.5 pyserial 3.4 python-apt 2.0.0+ubuntu0.20.4.5 python-dateutil 2.8.1 python-debian 0.1.36ubuntu1 pytz 2021.1 PyYAML 5.3.1 requests 2.22.0 requests-unixsocket 0.2.0 ruamel.yaml 0.17.9 ruamel.yaml.clib 0.2.2 SecretStorage 2.3.1 service-identity 18.1.0 setuptools 45.2.0 simplejson 3.16.0 six 1.14.0 sos 4.1 ssh-import-id 5.10 stix2 2.1.0 stix2-patterns 1.3.2 systemd-python 234 taxii2-client 2.3.0 tenacity 7.0.0 Twisted 18.9.0 ubuntu-advantage-tools 27.0 ufw 0.36 unattended-upgrades 0.1 urllib3 1.25.8 wadllib 1.3.3 wheel 0.34.2 XlsxWriter 1.4.3 zipp 1.0.0 zope.interface 4.7.1

rubinatorz commented 3 years ago

hi @hawki999

I noticed you are using attackcti==0.3.4.3, while in requirements.txt we set it to 0.3.3. Please try to install the python library versions as mentioned in the requirements.txt and try again. I tried with the 0.3.4.3 version and it gives me the same error, so pretty sure when you use the 0.3.3 version it should work.

We will look into the error for the 0.3.4.3 version for future releases.

hawki999 commented 3 years ago

Ruben

me bad ,

top man works perfect thanks for all your help have a good weekend

rubinatorz commented 3 years ago

Hi @hawki999

Great! My pleasure.

Enjoy the weekend!

rabobank-cdc / DeTTECT

Converting YAML to Json #54