TypeError: 'NoneType' object is not subscriptable

phucsnguyenv commented 4 years ago

[Connector] TypeError: 'NoneType' object is not subscriptable

Hi, I am creating a connector that import ipv4 from a file then create observable and indicators from that.

Here is folder list

.
├── config.yml
├── ip_list.txt
├── ip.py
└── test.py

My main code


import yaml
import os
import requests
import json
import wget
from pycti import OpenCTIConnectorHelper, get_config_variable

class Test: def init(self): config_file_path = os.path.dirname( os.path.abspath(file)) + "/config.yml" config = ( yaml.load(open(config_file_path), Loader=yaml.FullLoader) if os.path.isfile(config_file_path) else {} ) self.helper = OpenCTIConnectorHelper(config) self.max_tlp = get_config_variable("TALOSIP_MAX_TLP",["talosip", "max_tlp"], config)

def start(self):
    fo = open("ip_list.txt", "r")
    observable = []
    for ip in fo:
        ip = ip.strip("\n")
        ip = self.helper.api.stix_observable.create(
            type="ipv4-addr",
            observable_value=ip,
            markingDefinitions='TLP:WHITE',
            description="from talos via OPENCTI",
            createIndicator="True"
        )
    observable.append(ip["id"])
    print(observable)

a = Test() a.start()

- config
```yml
opencti:
    url: 'http://localhost'
    token: '<specified>'

connector:
    id: 'fc6c3837-aed2-4905-a525-c2294f8be358'
    type: 'EXTERNAL_IMPORT'
    name: 'IP blacklist'
    scope: 'ipv4-addr'
    confidence_level: 3
    log_level: 'info'
    interval: 7

talosip:
    max_tlp: 'TLP:AMBER'

Now when I run this code by $ python3 ip.py, it returns error

INFO:root:Listing Threat-Actors with filters null.
INFO:root:Starting ping alive thread
INFO:root:Listing StixObservables with filters [{"key": "observable_value", "values": ["141.8.226.58"]}].
INFO:root:Creating Stix-Observable {141.8.226.58} with indicator at True.
INFO:root:{"errors":[{"message":"An unknown error has occurred!  Please try again later.","name":"Unknown","time_thrown":"2020-05-04T06:34:07.701Z","data":{"type":"technical","level":"error"}}]}

Traceback (most recent call last):
  File "ip.py", line 39, in <module>
    a.start()
  File "ip.py", line 32, in start
    createIndicator="True"
  File "/home/sysadmin/.local/lib/python3.6/site-packages/pycti/entities/opencti_stix_observable.py", line 343, in create
    createIndicator=create_indicator,
  File "/home/sysadmin/.local/lib/python3.6/site-packages/pycti/entities/opencti_stix_observable.py", line 292, in create_raw
    result["data"]["stixObservableAdd"]
TypeError: 'NoneType' object is not subscriptable

Environment

OS (where OpenCTI server runs): Ubuntu 18.04
OpenCTI version: 3.1
OpenCTI client: python
Other environment details:

phucsnguyenv commented 4 years ago

Hi, I believe this error return from graphql

INFO:root:{"errors":[{"message":"An unknown error has occurred! Please try again later.","name":"Unknown","time_thrown":"2020-05-04T06:34:07.701Z","data":{"type":"technical","level":"error"}}]}

I tried to pull pycti 3.2.3 but the issue still persist. The code currently used

for ip in fo:
            ip = ip.strip("\n")
            ip = self.helper.api.stix_observable.create(
                type="ipv4-addr",
                observable_value=ip,
                createIndicator="True"
            )

Could you please take a look.

richard-julien commented 4 years ago

Hi @phucsnguyenv, can you check the logs on opencti side? Thanks

phucsnguyenv commented 4 years ago

hi @richard-julien

It's correct. Because if I use Indicator and Bundle method from stix2 library and send the bundle with helper.send_stix2_bundle, I can send the data

phucsnguyenv commented 4 years ago

I tried to use Playground on opencti web app and use the following mutation to test adding File-MD5

mutation stixObservableAdd($input: StixObservableAddInput){
  stixObservableAdd(input: $input){
    id
    entity_type
    observable_value
  }
},
{
  "input": {
    "internal_id_key": "996",
    "type":"File-MD5",
    "observable_value": "lfjawo93jlfjlfkjalwejflakw4r"
  }
}

It's working, but if I use the same mutation to add ipv4-addr, It failed with the following output. This output is as same as I run the python program.

mutation stixObservableAdd($input: StixObservableAddInput){
  stixObservableAdd(input: $input){
    id
    entity_type
    observable_value
  }
},
{
  "input": {
    "internal_id_key": "996",
    "type":"ipv4-addr",
    "observable_value": "192.168.1.1"
  }
}

{
  "errors": [
    {
      "message": "An unknown error has occurred!  Please try again later.",
      "name": "Unknown",
      "time_thrown": "2020-05-12T08:02:13.208Z",
      "data": {
        "type": "technical",
        "level": "error"
      }
    }
  ],
  "data": {
    "stixObservableAdd": null
  }
}

Following is GRAKN and OPENCTI log

GAKN

grakn_1                                  | 2020-05-12T08:02:13.636452676Z 2020-05-12 08:02:13,204 [transaction-listener] ERROR g.c.s.r.SessionService$TransactionListener - Runtime Exception in RPC TransactionListener: 
grakn_1                                  | 2020-05-12T08:02:13.636839423Z grakn.core.kb.graql.exception.GraqlSemanticException: ipv4-addr doesn't have an 'isa', a 'sub' or an 'id'

Here is log from OpenCTI

opencti_1                                | 2020-05-12T08:02:13.208676341Z {"code":3,"metadata":{"_internal_repr":{},"flags":0},"details":"GraqlSemanticException-ipv4-addr doesn't have an 'isa', a 'sub' or an 'id'. Please check server logs for the stack trace.","level":"error","message":"[GRAKN] executeWrite error > 3 INVALID_ARGUMENT: GraqlSemanticException-ipv4-addr doesn't have an 'isa', a 'sub' or an 'id'. Please check server logs for the stack trace.","stack":"Error: 3 INVALID_ARGUMENT: GraqlSemanticException-ipv4-addr doesn't have an 'isa', a 'sub' or an 'id'. Please check server logs for the stack trace.\n    at Object.exports.createStatusError (/opt/opencti/node_modules/grpc/src/common.js:91:15)\n    at ClientDuplexStream._emitStatusIfDone (/opt/opencti/node_modules/grpc/src/client.js:233:26)\n    at ClientDuplexStream._receiveStatus (/opt/opencti/node_modules/grpc/src/client.js:211:8)\n    at Object.onReceiveStatus (/opt/opencti/node_modules/grpc/src/client_interceptors.js:1311:15)\n    at InterceptingListener._callNext (/opt/opencti/node_modules/grpc/src/client_interceptors.js:568:42)\n    at InterceptingListener.onReceiveStatus (/opt/opencti/node_modules/grpc/src/client_interceptors.js:618:8)\n    at /opt/opencti/node_modules/grpc/src/client_interceptors.js:1127:18"}
opencti_1                                | 2020-05-12T08:02:13.209042422Z {"message":"[OPENCTI] Technical error > 3 INVALID_ARGUMENT: GraqlSemanticException-ipv4-addr doesn't have an 'isa', a 'sub' or an 'id'. Please check server logs for the stack trace.","locations":[{"line":2,"column":3}],"path":["stixObservableAdd"],"extensions":{"code":"INTERNAL_SERVER_ERROR","exception":{"code":3,"metadata":{"_internal_repr":{},"flags":0},"details":"GraqlSemanticException-ipv4-addr doesn't have an 'isa', a 'sub' or an 'id'. Please check server logs for the stack trace.","stacktrace":["Error: 3 INVALID_ARGUMENT: GraqlSemanticException-ipv4-addr doesn't have an 'isa', a 'sub' or an 'id'. Please check server logs for the stack trace.","    at Object.exports.createStatusError (/opt/opencti/node_modules/grpc/src/common.js:91:15)","    at ClientDuplexStream._emitStatusIfDone (/opt/opencti/node_modules/grpc/src/client.js:233:26)","    at ClientDuplexStream._receiveStatus (/opt/opencti/node_modules/grpc/src/client.js:211:8)","    at Object.onReceiveStatus (/opt/opencti/node_modules/grpc/src/client_interceptors.js:1311:15)","    at InterceptingListener._callNext (/opt/opencti/node_modules/grpc/src/client_interceptors.js:568:42)","    at InterceptingListener.onReceiveStatus (/opt/opencti/node_modules/grpc/src/client_interceptors.js:618:8)","    at /opt/opencti/node_modules/grpc/src/client_interceptors.js:1127:18"]}},"level":"error"}

Hope this help

richard-julien commented 4 years ago

Can you specified the version of OpenCTI you used? thanks

phucsnguyenv commented 4 years ago

Thanks for getting back.

Here is version of stack that I used OpenCTI: 3.0.3 GRAKN 1.6.2 Elasticsearch 7.5.2 RabbitMQ 3.7.17 Redis 5.0.8 Minio RELEASE.2019-10-12T01-39-57Z

richard-julien commented 4 years ago

Can you please upgrade to the latest openCTI version? Lot of bugs fixed since the 3.0.3. Thanks

phucsnguyenv commented 4 years ago

I just need to pull new docker image right? Will my data be persisted?

phucsnguyenv commented 4 years ago

I updated the latest version of OpenCTI and here is the outcome GrapQL

mutation stixObservableAdd($input: StixObservableAddInput){
  stixObservableAdd(input: $input){
    entity_type
    observable_value
  }
},
{
  "input": {
    "type":"ipv4-addr",
    "observable_value": "192.168.1.1"
  }
}

{
  "errors": [
    {
      "message": "An unknown error has occurred!  Please try again later.",
      "name": "Unknown",
      "time_thrown": "2020-05-14T04:05:51.564Z",
      "data": {
        "type": "technical",
        "level": "error"
      }
    }
  ],
  "data": {
    "stixObservableAdd": null
  }
}

opencti_1                                | 2020-05-14T04:07:32.001734480Z {"error":{"message":"Variable \"$input\" got invalid value [\"\"] at \"input.secondary_motivation\"; Expected type String. String cannot represent a non string value: [\"\"]","locations":[{"line":2,"column":42}],"extensions":{"code":"INTERNAL_SERVER_ERROR","exception":{"stacktrace":["TypeError: String cannot represent a non string value: [\"\"]","    at GraphQLScalarType.coerceString [as parseValue] (/opt/opencti/node_modules/graphql/type/scalars.js:164:11)","    at coerceInputValueImpl (/opt/opencti/node_modules/graphql/utilities/coerceInputValue.js:127:26)","    at coerceInputValueImpl (/opt/opencti/node_modules/graphql/utilities/coerceInputValue.js:105:35)","    at coerceInputValue (/opt/opencti/node_modules/graphql/utilities/coerceInputValue.js:37:10)","    at _loop (/opt/opencti/node_modules/graphql/execution/values.js:107:69)","    at coerceVariableValues (/opt/opencti/node_modules/graphql/execution/values.js:119:16)","    at getVariableValues (/opt/opencti/node_modules/graphql/execution/values.js:48:19)","    at buildExecutionContext (/opt/opencti/node_modules/graphql/execution/execute.js:184:61)","    at executeImpl (/opt/opencti/node_modules/graphql/execution/execute.js:89:20)","    at Object.execute (/opt/opencti/node_modules/graphql/execution/execute.js:64:35)","    at /opt/opencti/node_modules/apollo-server-core/dist/requestPipeline.js:246:46","    at Generator.next (<anonymous>)","    at /opt/opencti/node_modules/apollo-server-core/dist/requestPipeline.js:8:71","    at new Promise (<anonymous>)","    at __awaiter (/opt/opencti/node_modules/apollo-server-core/dist/requestPipeline.js:4:12)","    at execute (/opt/opencti/node_modules/apollo-server-core/dist/requestPipeline.js:225:20)"]}}},"level":"error","message":"[OPENCTI] Technical error"}
opencti_1                                | 2020-05-14T04:07:33.097116171Z {"error":{"message":"Variable \"$input\" got invalid value [\"\"] at \"input.secondary_motivation\"; Expected type String. String cannot represent a non string value: [\"\"]","locations":[{"line":2,"column":42}],"extensions":{"code":"INTERNAL_SERVER_ERROR","exception":{"stacktrace":["TypeError: String cannot represent a non string value: [\"\"]","    at GraphQLScalarType.coerceString [as parseValue] (/opt/opencti/node_modules/graphql/type/scalars.js:164:11)","    at coerceInputValueImpl (/opt/opencti/node_modules/graphql/utilities/coerceInputValue.js:127:26)","    at coerceInputValueImpl (/opt/opencti/node_modules/graphql/utilities/coerceInputValue.js:105:35)","    at coerceInputValue (/opt/opencti/node_modules/graphql/utilities/coerceInputValue.js:37:10)","    at _loop (/opt/opencti/node_modules/graphql/execution/values.js:107:69)","    at coerceVariableValues (/opt/opencti/node_modules/graphql/execution/values.js:119:16)","    at getVariableValues (/opt/opencti/node_modules/graphql/execution/values.js:48:19)","    at buildExecutionContext (/opt/opencti/node_modules/graphql/execution/execute.js:184:61)","    at executeImpl (/opt/opencti/node_modules/graphql/execution/execute.js:89:20)","    at Object.execute (/opt/opencti/node_modules/graphql/execution/execute.js:64:35)","    at /opt/opencti/node_modules/apollo-server-core/dist/requestPipeline.js:246:46","    at Generator.next (<anonymous>)","    at /opt/opencti/node_modules/apollo-server-core/dist/requestPipeline.js:8:71","    at new Promise (<anonymous>)","    at __awaiter (/opt/opencti/node_modules/apollo-server-core/dist/requestPipeline.js:4:12)","    at execute (/opt/opencti/node_modules/apollo-server-core/dist/requestPipeline.js:225:20)"]}}},"level":"error","message":"[OPENCTI] Technical error"}

phucsnguyenv commented 4 years ago

I think I have to type IPv4-Addr instead of ipv4-addr. It's case-sensitive, isn't it?

richard-julien commented 4 years ago

secondary_motivation is not a field existing in StixObservableAddInput. Are you sure this error is related to your stixObservableAdd call? Could you clear your logs and test again? Thanks

phucsnguyenv commented 4 years ago

hmmm, I am sure this log I get is after execute the stixObserbaleAdd call. I try again

for ip in fo:
            ip = ip.strip("\n")
            ip = self.helper.api.stix_observable.create(
                type="IPv4-Addr",
                observable_value=ip,
                createIndicator="True"
            )

Changing ipv4-addr to IPv4-Addr, and it works for me. I tried to test with GraphQL directly and the same issue happened. So the solution for me is reformatting the type of observable correctly before sending it.

Really thanks for your help

SamuelHassine commented 4 years ago

Hello,

We clearly lack of documentation in this case and we are sorry for that. There are 2 problems in your source code, the TLP and the createIndicator (Boolean).

For the TLP, you have to pass a list of TLP OpenCTI IDs. You can get them by using the official TLP STIX ID or by searching them directly:

import TLP_WHITE from stix2
tlp_white_marking_definition = self.helper.api.marking_definition.read(id=TLP_WHITE["id"])
// or
tlp_white_marking_definition = self.helper.api.marking_definition.read(filters={"key": "definition", "values": ["TLP:WHITE"]})

Then:

ip = self.helper.api.stix_observable.create(
  type="ipv4-addr",
  observable_value=ip,
  markingDefinitions=[tlp_white_marking_definition['id']],
  description="from talos via OPENCTI",
  createIndicator=True
)

phucsnguyenv commented 4 years ago

Many thanks,

It's very clear, I will try this approach. I just follow some codes from existing connector so it has some bugs at the first place.

SamuelHassine commented 4 years ago

Yes, these bugs have been solved now ;)

SamuelHassine commented 4 years ago

Is this working now?

phucsnguyenv commented 4 years ago

Many thanks,

I have not tried yet, but I believe this would work. As I can see the problems in my code :D. I will try this next week.

I think we ca close this issue here...

Thanks again.

OpenCTI-Platform / connectors

TypeError: 'NoneType' object is not subscriptable #125

[Connector] TypeError: 'NoneType' object is not subscriptable

Environment