Docker container doesn't start with described collections/services/etc from data-configuration.yml

teuber789 commented 5 years ago

Hey all,

I've run into an interesting problem. The default docker container on port 9000 doesn't seem to have to default collections/services/etc described in the documentation.

Problem overview

When I run the default docker container, it seems to start up without any errors:

$ sudo docker run -p 9000:9000 eclecticiq/opentaxii
Using config:
---

domain: "localhost:9000"

persistence_api:
  class: opentaxii.persistence.sqldb.SQLDatabaseAPI
  parameters:
    db_connection: sqlite:////data/data.db
    create_tables: yes

auth_api:
  class: opentaxii.auth.sqldb.SQLDatabaseAPI
  parameters:
    db_connection: sqlite:////data/auth.db
    create_tables: yes
    secret: notVerySecret

logging:
  opentaxii: info
  root: info

hooks:
{"api": "opentaxii.persistence.sqldb.SQLDatabaseAPI", "event": "api.initialized", "logger": "opentaxii.utils", "level": "info", "timestamp": "2019-09-12T15:34:52.206834Z"}
{"api": "opentaxii.auth.sqldb.SQLDatabaseAPI", "event": "api.initialized", "logger": "opentaxii.utils", "level": "info", "timestamp": "2019-09-12T15:34:52.249371Z"}
{"event": "opentaxii.server_configured", "logger": "opentaxii.server", "level": "info", "timestamp": "2019-09-12T15:34:52.249701Z"}
{"event": "Starting gunicorn 19.9.0", "logger": "gunicorn.error", "level": "info", "timestamp": "2019-09-12T15:34:52.285789Z"}
{"event": "Listening at: http://0.0.0.0:9000 (1)", "logger": "gunicorn.error", "level": "info", "timestamp": "2019-09-12T15:34:52.288185Z"}
{"event": "Using worker: sync", "logger": "gunicorn.error", "level": "info", "timestamp": "2019-09-12T15:34:52.288568Z"}
{"event": "Booting worker with pid: 12", "logger": "gunicorn.error", "level": "info", "timestamp": "2019-09-12T15:34:52.292445Z"}
{"event": "Booting worker with pid: 13", "logger": "gunicorn.error", "level": "info", "timestamp": "2019-09-12T15:34:52.376715Z"}

However, I get a 404 when trying to poll one of the default collections:

$ taxii-poll --host localhost --port 9000 --discovery /services/discovery-a --collection collection-a
2019-09-12 11:42:06,136 INFO: Polling using data binding: ALL
2019-09-12 11:42:06,138 INFO: Sending Discovery_Request to http://localhost:9000/services/discovery-a
2019-09-12 11:42:06,152 ERROR: Can not autodiscover advertised services
2019-09-12 11:42:06,152 ERROR: HTTP Error: status code 404

Same for discovery:

$ taxii-discovery --host localhost --port 9000 --discovery services/discovery-a
2019-09-12 11:45:14,495 INFO: Sending Discovery_Request to http://localhost:9000/services/discovery-a
2019-09-12 11:45:14,561 ERROR: HTTP Error: status code 404

It looks like none of the entities were copied over from the data-configuration.yml file:

[admin@localhost stuff]$ sudo docker exec -it 024d810152d6 "/bin/bash"

root@024d810152d6:/# apt update
<output omitted>

root@024d810152d6:/# apt install sqlite3
<output omitted>

root@024d810152d6:/# sqlite3 /data/data.db
SQLite version 3.16.2 2017-01-06 16:32:41
Enter ".help" for usage hints.

sqlite> .tables
collection_to_content_block  result_sets
content_blocks               service_to_collection
data_collections             services
inbox_messages               subscriptions

sqlite> select * from services;

sqlite> select count(*) from services;
0

sqlite> select * from data_collections;

sqlite> select count(*) from data_collections;
0
sqlite> .exit

Setup

OS: CentOS 7.6
Docker version 19.03.2
Cabby version: 0.1.20 (latest)

Any ideas?

teuber789 commented 5 years ago

Is this a regression? It appears to have been fixed at one point: https://github.com/eclecticiq/OpenTAXII/issues/127

teuber789 commented 5 years ago

@arcsector @traut you two often frequent this project. This issue is still happening. Any ideas?

arcsector commented 4 years ago

@jtcotton63 Does this happen when you build the image from the dockerfile in a clone of the repo as well?

eric-eclecticiq commented 3 years ago

The bug is still present in the Docker image in the hub.

Does this happen when you build the image from the dockerfile in a clone of the repo as well?

Yes

Workaround

docker run \
    -p 9000:9000 \
    -v /path/to/local/repo/examples/data-configuration.yml:/input/data-configuration.yml \
    eclecticiq/opentaxii

More in the documentation: https://opentaxii.readthedocs.io/en/stable/docker.html#volumes

Regression

As the README says

NOTE: OpenTAXII is now accessible through port 9000, with data stored locally in a SQLite databases using services/collections/accounts defined in data-configuration.

eric-eclecticiq commented 3 years ago

It is indeed a bug and we’d love to have it fixed, however it’s not a high priority for our team at the moment, so we can’t promise when it will get fixed. Still, we’re very open to external contributions - if you know how to fix this problem and you can open a PR with a fix, we will be very grateful.

erwin-eiq commented 3 years ago

I think this is a documentation issue instead of a bug, because of these reasons:

127 mentions cd to 'examples' folder under local git repo. Mount the folder into input directory while starting the opentaxii container.
The example docker-compose.yml also mounts the same directory into /input
If we put a default file in the image, it would load it every time you restart the container. That doesn't sound like desired behaviour.

Hence, I will fix the documentation instead.

eclecticiq / OpenTAXII