500 response while hitting the api taxii-discovery --path http://localhost:9000/services/discovery-a

[rohits144]

I tried to start the the dev server with the command - "opentaxii-run-dev" the tried the below command :

taxii-discovery --path http://localhost:9000/services/discovery-a

i get this response :

2021-12-08 13:32:23,379 INFO: Sending Discovery_Request to http://localhost:9000/services/discovery-a 2021-12-08 13:32:23,421 ERROR: HTTP Error: status code 500

in the dev server log i can see these logs:

2021-12-08T07:53:16.353670Z [werkzeug] info: 127.0.0.1 - - [08/Dec/2021 13:23:16] "POST /management/auth HTTP/1.1" 500 - Traceback (most recent call last): File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1702, in _execute_context context = constructor( File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1000, in _init_compiled self.cursor = self.create_cursor() File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1348, in create_cursor return self.create_default_cursor() File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1351, in create_default_cursor return self._dbapi_connection.cursor() File "/usr/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 1083, in cursor return self.dbapi_connection.cursor(*args, **kwargs) sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140278871557952 and this is thread id 140278732097280.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/lib/python3.8/site-packages/flask/app.py", line 2091, in call return self.wsgi_app(environ, start_response) File "/usr/lib/python3.8/site-packages/flask/app.py", line 2076, in wsgi_app response = self.handle_exception(e) File "/usr/lib/python3.8/site-packages/flask/app.py", line 2073, in wsgi_app response = self.full_dispatch_request() File "/usr/lib/python3.8/site-packages/flask/app.py", line 1518, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/lib/python3.8/site-packages/flask/app.py", line 1516, in full_dispatch_request rv = self.dispatch_request() File "/usr/lib/python3.8/site-packages/flask/app.py", line 1502, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(req.view_args) File "/usr/lib/python3.8/site-packages/opentaxii/management.py", line 18, in auth token = context.server.auth.authenticate(username, password) File "/usr/lib/python3.8/site-packages/opentaxii/auth/manager.py", line 29, in authenticate return self.api.authenticate(username, password) File "/usr/lib/python3.8/site-packages/opentaxii/auth/sqldb/api.py", line 58, in authenticate account = Account.query.filter_by(username=username).one() File "/usr/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2856, in one return self._iter().one() File "/usr/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2894, in _iter result = self.session.execute( File "/usr/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1689, in execute result = conn._execute_20(statement, params or {}, execution_options) File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1614, in _execute_20 return meth(self, args_10style, kwargs_10style, execution_options) File "/usr/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection return connection._execute_clauseelement( File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1481, in _execute_clauseelement ret = self._execute_context( File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1708, in _execute_context self._handle_dbapi_exception( File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2026, in _handle_dbapiexception util.raise( File "/usr/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_ raise exception File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1702, in _execute_context context = constructor( File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1000, in _init_compiled self.cursor = self.create_cursor() File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1348, in create_cursor return self.create_default_cursor() File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1351, in create_default_cursor return self._dbapi_connection.cursor() File "/usr/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 1083, in cursor return self.dbapi_connection.cursor(*args, *kwargs) sqlalchemy.exc.ProgrammingError: (sqlite3.ProgrammingError) SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140278871557952 and this is thread id 140278732097280. [SQL: SELECT accounts.id AS accounts_id, accounts.username AS accounts_username, accounts.password_hash AS accounts_password_hash, accounts.is_admin AS accounts_is_admin, accounts._permissions AS accounts__permissions FROM accounts WHERE accounts.username = ?] [parameters: [{}]] (Background on this error at: https://sqlalche.me/e/14/f405) 2021-12-08T08:02:23.420169Z [werkzeug] info: 127.0.0.1 - - [08/Dec/2021 13:32:23] "POST /services/discovery-a HTTP/1.1" 500 - Traceback (most recent call last): File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1702, in _execute_context context = constructor( File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1000, in _init_compiled self.cursor = self.create_cursor() File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1348, in create_cursor return self.create_default_cursor() File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1351, in create_default_cursor return self._dbapi_connection.cursor() File "/usr/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 1083, in cursor return self.dbapi_connection.cursor(args, kwargs) sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140278871557952 and this is thread id 140278732097280.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/lib/python3.8/site-packages/flask/app.py", line 2091, in call return self.wsgi_app(environ, start_response) File "/usr/lib/python3.8/site-packages/flask/app.py", line 2076, in wsgi_app response = self.handle_exception(e) File "/usr/lib/python3.8/site-packages/flask/app.py", line 2073, in wsgi_app response = self.full_dispatch_request() File "/usr/lib/python3.8/site-packages/flask/app.py", line 1518, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/lib/python3.8/site-packages/flask/app.py", line 1516, in full_dispatch_request rv = self.dispatch_request() File "/usr/lib/python3.8/site-packages/flask/app.py", line 1502, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(*req.view_args) File "/usr/lib/python3.8/site-packages/opentaxii/middleware.py", line 69, in wrapper services = server.get_services() File "/usr/lib/python3.8/site-packages/opentaxii/server.py", line 107, in get_services service_entities = self.persistence.get_services() File "/usr/lib/python3.8/site-packages/opentaxii/persistence/manager.py", line 90, in get_services return self.api.get_services() File "/usr/lib/python3.8/site-packages/opentaxii/persistence/sqldb/api.py", line 59, in get_services services = Service.query.all() File "/usr/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2759, in all return self._iter().all() File "/usr/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2894, in _iter result = self.session.execute( File "/usr/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1689, in execute result = conn._execute_20(statement, params or {}, execution_options) File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1614, in _execute_20 return meth(self, args_10style, kwargs_10style, execution_options) File "/usr/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection return connection._execute_clauseelement( File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1481, in _execute_clauseelement ret = self._execute_context( File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1708, in _execute_context self._handle_dbapi_exception( File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2026, in _handle_dbapiexception util.raise( File "/usr/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_ raise exception File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1702, in _execute_context context = constructor( File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1000, in _init_compiled self.cursor = self.create_cursor() File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1348, in create_cursor return self.create_default_cursor() File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 1351, in create_default_cursor return self._dbapi_connection.cursor() File "/usr/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 1083, in cursor return self.dbapi_connection.cursor(args, **kwargs) sqlalchemy.exc.ProgrammingError: (sqlite3.ProgrammingError) SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140278871557952 and this is thread id 140278732097280. [SQL: SELECT services.date_created AS services_date_created, services.id AS services_id, services.type AS services_type, services._properties AS services__properties, services.date_updated AS services_date_updated FROM services] [parameters: [{}]] (Background on this error at: https://sqlalche.me/e/14/f405)

can you please let me know if this is expected behavior and how to fix this. i am using WSL ubuntu to run this.

Note: - the heath check command (curl http://localhost:9000/management/health) works fine without issue

[erwin-eiq]

Hi @rohits144 thanks for this bugreport.

I've reproduced it and apparently I broke something between 0.2.0 and 0.2.1. I can suggest going back to 0.2.0 as a temporary workaround until I get this fixed. I'll get started on a fix.

[rohits144]

Hi @erwin-eiq, Thanks for your response. One of my colleague helped me with a workaround which is working for me. We need to add ("check_same_thread=False") this part in the default.yml config file at db_connector. kindly see below

--

domain: "localhost:9000"

support_basic_auth: yes

save_raw_inbox_messages: yes

xml_parser_supports_huge_tree: yes

count_blocks_in_poll_responses: no

return_server_error_details: no

unauthorized_status: UNAUTHORIZED

persistence_api:

class: opentaxii.persistence.sqldb.SQLDatabaseAPI

parameters:

db_connection: sqlite:////tmp/data.db?check_same_thread=False

create_tables: yes

auth_api:

class: opentaxii.auth.sqldb.SQLDatabaseAPI

parameters:

db_connection: sqlite:////tmp/auth.db?check_same_thread=False

create_tables: yes

secret: TEST_SECRET

token_ttl_secs: 3600

logging:

opentaxii: info

root: info

hooks:

---

Adding that solves the error for me.

[erwin-eiq]

Thanks for the additional information and the suggested workaround. However, this workaround is dangerous according to both the python docs and sqlalchemy docs.

Python docs (https://docs.python.org/3/library/sqlite3.html):

By default, check_same_thread is True and only the creating thread may use the
connection. If set False, the returned connection may be shared across multiple
threads. When using multiple threads with the same connection writing operations
should be serialized by the user to avoid data corruption.

Sqlalchemy docs (https://docs.sqlalchemy.org/en/14/dialects/sqlite.html#threading-pooling-behavior)

Pysqlite does include a now-undocumented flag known as check_same_thread which
will disable this check, however note that pysqlite connections are still not
safe to use in concurrently in multiple threads. In particular, any statement
execution calls would need to be externally mutexed, as Pysqlite does not
provide for thread-safe propagation of error messages among other things.

I've got a proof-of-concept fix ready that doesn't need check_same_thread=False, just need to clean it up properly and will push it to a bugfix release soon.

[erwin-eiq]

@rohits144 I've released 0.2.3 just now with a fix for your issue.