Debug threat object in feeds exception error

[priamai]

I am looking into the error to see if is PIP related or code related. @andrewgin and @dfjosullivan

[priamai]

@dfjosullivan and @andrewgin can you try to run this code: https://github.com/os-threat/os-intel/blob/dev/feeds/phishing-database/src/main.py with python main.py -i

I am getting this error and seems to be related to GPRC

/home/robomotic/DevOps/github/os-intel/venv/bin/python /home/robomotic/DevOps/github/os-intel/feeds/phishing-database/src/main.py -i -r 
2023-06-21 11:14:52.449 | DEBUG    | stixorm.module.definitions.stix21:<module>:171 - Loaded 135 stix dictionary objects
2023-06-21 11:14:52.456 | DEBUG    | stixorm.module.definitions.attack:<module>:120 - Loaded 39 attack dictionary objects
2023-06-21 11:14:52.460 | DEBUG    | stixorm.module.definitions.os_threat:<module>:94 - Loaded 14 os-threat dictionary objects
2023-06-21 11:14:52.461 | DEBUG    | stixorm.module.definitions.cacao:<module>:84 - Loaded 1 cacao dictionary objects
2023-06-21 11:14:52.461 | DEBUG    | stixorm.module.definitions.kestrel:<module>:83 - Loaded 1 kestrel dictionary objects
INFO:__main__:Init database
ERROR:root:Traceback (most recent call last):

  File "/home/robomotic/DevOps/github/os-intel/venv/lib/python3.10/site-packages/returns/result.py", line 561, in decorator
    return Success(inner_function(*args, **kwargs))

  File "/home/robomotic/DevOps/github/os-intel/venv/lib/python3.10/site-packages/stixorm/module/typedb.py", line 110, in __initialise
    setup_database(self._stix_connection, self.clear)

  File "/home/robomotic/DevOps/github/os-intel/venv/lib/python3.10/site-packages/stixorm/module/initialise.py", line 55, in setup_database
    with TypeDB.core_client(url) as client:

  File "/home/robomotic/DevOps/github/os-intel/venv/lib/python3.10/site-packages/typedb/client.py", line 70, in core_client
    return _CoreClient(address, parallelisation)

  File "/home/robomotic/DevOps/github/os-intel/venv/lib/python3.10/site-packages/typedb/connection/core/client.py", line 35, in __init__
    self._channel, self._stub = self.new_channel_and_stub()

  File "/home/robomotic/DevOps/github/os-intel/venv/lib/python3.10/site-packages/typedb/connection/core/client.py", line 52, in new_channel_and_stub
    return channel, _CoreStub(channel)

  File "/home/robomotic/DevOps/github/os-intel/venv/lib/python3.10/site-packages/typedb/connection/core/stub.py", line 42, in __init__
    raise e

  File "/home/robomotic/DevOps/github/os-intel/venv/lib/python3.10/site-packages/typedb/connection/core/stub.py", line 39, in __init__
    self._stub.connection_open(connection_open_req())

  File "/home/robomotic/DevOps/github/os-intel/venv/lib/python3.10/site-packages/grpc/_channel.py", line 1030, in __call__
    return _end_unary_response_blocking(state, call, False, None)

  File "/home/robomotic/DevOps/github/os-intel/venv/lib/python3.10/site-packages/grpc/_channel.py", line 910, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.UNIMPLEMENTED
    details = "Method not found: typedb.protocol.TypeDB/connection_open"
    debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2023-06-21T11:14:52.75518366+01:00", grpc_status:12, grpc_message:"Method not found: typedb.protocol.TypeDB/connection_open"}"
>
NoneType: None
Traceback (most recent call last):
  File "/home/robomotic/DevOps/github/os-intel/feeds/phishing-database/src/main.py", line 286, in <module>
    loader.init_feeds()
  File "/home/robomotic/DevOps/github/os-intel/feeds/phishing-database/src/main.py", line 228, in init_feeds
    typedb_sink = TypeDBSink(connection,clear_all, import_type)
  File "/home/robomotic/DevOps/github/os-intel/venv/lib/python3.10/site-packages/stixorm/module/typedb.py", line 95, in __init__
    raise Exception(str(result.failure()))
Exception: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.UNIMPLEMENTED
    details = "Method not found: typedb.protocol.TypeDB/connection_open"
    debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2023-06-21T11:14:52.75518366+01:00", grpc_status:12, grpc_message:"Method not found: typedb.protocol.TypeDB/connection_open"}"
>

Process finished with exit code 1

[priamai]

My setup: typedb-client=2.18.0 typedb-protocol=2.18.0 typedb=2.18.0

[priamai]

The problem seems to relate to the Init phase @dfjosullivan

[andrewgin]

I get the following:

2023-06-22 21:47:18.966 | DEBUG    | stixorm.module.definitions.stix21:<module>:171 - Loaded 135 stix dictionary objects
2023-06-22 21:47:18.987 | DEBUG    | stixorm.module.definitions.attack:<module>:120 - Loaded 39 attack dictionary objects
2023-06-22 21:47:19.001 | DEBUG    | stixorm.module.definitions.os_threat:<module>:94 - Loaded 14 os-threat dictionary objects
2023-06-22 21:47:19.003 | DEBUG    | stixorm.module.definitions.cacao:<module>:84 - Loaded 1 cacao dictionary objects
2023-06-22 21:47:19.005 | DEBUG    | stixorm.module.definitions.kestrel:<module>:83 - Loaded 1 kestrel dictionary objects
INFO:__main__:Init database
INFO:stixorm.module.initialise:========================== Database initialisation ============================
INFO:stixorm.module.initialise:typedb response ->
[_0/_StringAttribute[stix-type:0x836f80dc2800126d61726b696e672d646566696e6974696f6e]][_1/_StringAttribute[stix-id:0x836f80ea2800386d61726b696e672d646566696e6974696f6e2d2d36313366326532362d343037642d343863372d396563612d623865393164663939646339]][_2/_StringAttribute[spec-version:0x836f8016280003322e31]][_3/_DateTimeAttribute[created:0x836f80f73280000159b92c3800]][mark/_Entity[tlp-white:0x826e80337fffffffffffffff]]
INFO:stixorm.module.initialise:typedb response ->
[_0/_StringAttribute[stix-type:0x836f80dc2800126d61726b696e672d646566696e6974696f6e]][_1/_StringAttribute[stix-id:0x836f80ea2800386d61726b696e672d646566696e6974696f6e2d2d33343039386663652d383630662d343861652d386535302d656264336363356534316461]][_2/_StringAttribute[spec-version:0x836f8016280003322e31]][_3/_DateTimeAttribute[created:0x836f80f73280000159b92c3800]][mark/_Entity[tlp-green:0x826e80037fffffffffffffff]]
INFO:stixorm.module.initialise:typedb response ->
[_0/_StringAttribute[stix-type:0x836f80dc2800126d61726b696e672d646566696e6974696f6e]][_1/_StringAttribute[stix-id:0x836f80ea2800386d61726b696e672d646566696e6974696f6e2d2d66383864333166362d343836662d343464612d623331372d303133333362646530623832]][_2/_StringAttribute[spec-version:0x836f8016280003322e31]][_3/_DateTimeAttribute[created:0x836f80f73280000159b92c3800]][mark/_Entity[tlp-amber:0x826e801a7fffffffffffffff]]
INFO:stixorm.module.initialise:typedb response ->
[_0/_StringAttribute[stix-type:0x836f80dc2800126d61726b696e672d646566696e6974696f6e]][_1/_StringAttribute[stix-id:0x836f80ea2800386d61726b696e672d646566696e6974696f6e2d2d35653537633733392d333931612d346562332d623662652d376431356361393264356564]][_2/_StringAttribute[spec-version:0x836f8016280003322e31]][_3/_DateTimeAttribute[created:0x836f80f73280000159b92c3800]][mark/_Entity[tlp-red:0x826e800e7fffffffffffffff]]
INFO:stixorm.module.initialise:===============================================================================

INFO:stixorm.module.typedb:we have loaded Stix schema
INFO:stixorm.module.typedb:we have loaded Stix rules
INFO:stixorm.module.typedb:we have loaded ATT&CK schema
INFO:stixorm.module.typedb:we have loaded os-threat schema
INFO:stixorm.module.typedb:
##########################################################################################################################################################

INFO:stixorm.module.typedb_lib.queries:

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------- Query ids ------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

INFO:stixorm.module.typedb_lib.queries:match $id isa stix-id;$id "marking-definition--f74d1a96-c0cf-468c-bc66-152129d48e42";
INFO:stixorm.module.typedb_lib.queries:

INFO:__main__:Status = ResultStatus.MISSING_DEPENDENCY ID = feeds--78d04d18-de7e-4545-a59f-239022c68339 Error = None```

[dfjosullivan]

My results

2023-06-23 09:59:39.146 | DEBUG    | stixorm.module.definitions.stix21:<module>:171 - Loaded 135 stix dictionary objects
2023-06-23 09:59:39.164 | DEBUG    | stixorm.module.definitions.attack:<module>:120 - Loaded 39 attack dictionary objects
2023-06-23 09:59:39.179 | DEBUG    | stixorm.module.definitions.os_threat:<module>:94 - Loaded 14 os-threat dictionary objects
2023-06-23 09:59:39.181 | DEBUG    | stixorm.module.definitions.cacao:<module>:84 - Loaded 1 cacao dictionary objects
2023-06-23 09:59:39.184 | DEBUG    | stixorm.module.definitions.kestrel:<module>:83 - Loaded 1 kestrel dictionary objects
INFO:__main__:Init database
ERROR:root:Traceback (most recent call last):

  File "C:\Users\denis\PycharmProjects\os-intel\venv\lib\site-packages\returns\result.py", line 561, in decorator
    return Success(inner_function(*args, **kwargs))

  File "C:\Users\denis\PycharmProjects\os-intel\venv\lib\site-packages\stixorm\module\typedb.py", line 110, in __initialise
    setup_database(self._stix_connection, self.clear)

  File "C:\Users\denis\PycharmProjects\os-intel\venv\lib\site-packages\stixorm\module\initialise.py", line 55, in setup_database
    with TypeDB.core_client(url) as client:

  File "C:\Users\denis\PycharmProjects\os-intel\venv\lib\site-packages\typedb\client.py", line 70, in core_client
    return _CoreClient(address, parallelisation)

  File "C:\Users\denis\PycharmProjects\os-intel\venv\lib\site-packages\typedb\connection\core\client.py", line 35, in __init__
    self._channel, self._stub = self.new_channel_and_stub()

  File "C:\Users\denis\PycharmProjects\os-intel\venv\lib\site-packages\typedb\connection\core\client.py", line 52, in new_channel_and_stub
    return channel, _CoreStub(channel)

  File "C:\Users\denis\PycharmProjects\os-intel\venv\lib\site-packages\typedb\connection\core\stub.py", line 42, in __init__
    raise e

  File "C:\Users\denis\PycharmProjects\os-intel\venv\lib\site-packages\typedb\connection\core\stub.py", line 39, in __init__
    self._stub.connection_open(connection_open_req())

  File "C:\Users\denis\PycharmProjects\os-intel\venv\lib\site-packages\grpc\_channel.py", line 1030, in __call__
    return _end_unary_response_blocking(state, call, False, None)

  File "C:\Users\denis\PycharmProjects\os-intel\venv\lib\site-packages\grpc\_channel.py", line 910, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNIMPLEMENTED
        details = "Method not found: typedb.protocol.TypeDB/connection_open"
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2023-06-22T23:59:39.8546134+00:00", grpc_status:12, grpc_message:"Method not found: typedb.protocol.TypeDB/connection_open"}"
>
NoneType: None
Traceback (most recent call last):
  File "C:\Users\denis\PycharmProjects\os-intel\feeds\phishing-database\src\main.py", line 286, in <module>
    loader.init_feeds()
  File "C:\Users\denis\PycharmProjects\os-intel\feeds\phishing-database\src\main.py", line 228, in init_feeds
    typedb_sink = TypeDBSink(connection,clear_all, import_type)
  File "C:\Users\denis\PycharmProjects\os-intel\venv\lib\site-packages\stixorm\module\typedb.py", line 95, in __init__
    raise Exception(str(result.failure()))
Exception: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNIMPLEMENTED
        details = "Method not found: typedb.protocol.TypeDB/connection_open"
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2023-06-22T23:59:39.8546134+00:00", grpc_status:12, grpc_message:"Method not found: typedb.protocol.TypeDB/connection_open"}"
>

[priamai]

Are you using the pip test version or the latest code ?

[priamai]

Seems like Andrew went further in the steps!

[dfjosullivan]

pip test version

[priamai]

@andrewgin what version are you using for the package? did you create a new one? did you figure out what is that error about missing FEED ID?

[andrewgin]

I'm also using pip test version 0.1.3a0. I haven't looked into the feed id error yet. Have you run the stix orm tests locally? If those work then hopefully it means we can rule out an issue with that and it means its either with your typedb setup or something in os-intel main.py

[priamai]

I have uninstalled and re-installed the same version but I am getting the same error which is different from yours. I have a fresh venv and fresh docker image for Vaticle so I am not sure what can be different. I don't see any tests in this current repository so I am not sure what should I run. Let's do a screenshare tomorrow or Monday. Cheers!

[andrewgin]

fyi I'm also using

typedb-client==2.18.0
typedb-protocol==2.18.0

typedb server --version also says 2.18