Add multi-domain support

ropetin commented 2 years ago

(edit by @erwin-eiq) This question has been turned into a feature request. Description in comment

(original question)

Hopefully I'm not missing something obvious in the documentation, but I can't find a definitive answer for this question. Is it possible to set a single OpenTAXII server with multiple OPENTAXXI_DOMAIN / domain settings? I need to have the same data available from multiple endpoints on different IPs, ports, and domain names. I can route the traffic fine with my firewall and HAProxy, but obviously that doesn't help changing the URLs sent back with poll requests, for example.

arcsector commented 2 years ago

Hi @ropetin,

Are you deploying on docker or a VM? Regardless, you can set the opentaxii variable to 0.0.0.0 and the server will listen on whatever name it requires. This is even more trivial when you utilize an http proxy in order to listen on a specific name. The web architecture for a distributed deployment would have to be coordinated by yourself

ropetin commented 2 years ago

I'm using Docker, and I'm specifically referring to the ENV variable, OPENTAXII_DOMAIN. Whatever I set that to is what gets returned when doing a poll request, for example. I can set it to 0.0.0.0 or 0.0.0.0:9000, but poll requests will fail. For example, I just set it to 0.0.0.0:9000, restarted the container, and now when I connect with: taxii-poll --host my.domain.name .... I get back the following, which obviously fails. I have to be missing something obvious, but I can't figure out what!

2021-10-16 14:04:06,663 INFO: Sending Poll_Request to https://0.0.0.0:9000/services/poll

arcsector commented 2 years ago

Did you try using localhost as the host instead? If you can't reach it via localhost, you're probably not binding docker to the port correctly, or you have a firewall blocking that port from connecting to the host. In addition, using https when, as far as I know, OpenTAXII doesn't support SSL OOTB, is going to be a futile protocol mismatch - you should try http first

In either case, best practice is to use a web proxy in order to sit in front of OpenTAXII and forward to it, so that you can control SSL and Cipher configs from there instead of in flask. Can you try that as well?

ErwinJunge commented 2 years ago

I think @ropetin is talking about the service addresses in the output including the configured domain and wanting to have multiple domain support there.

If so, the current answer is "no, or at least not easily". It's not supported directly in the opentaxxi code. To achieve the goal, there are essentially 2 options: 1) Set up 2 instances of opentaxii with the same db connection parameters, but different domain settings. Then use your reverse proxy to select the correct one. This is achievable now, but obviously a bit wasteful on resources. 2) Build host-header checking into opentaxii to detect the hostname at runtime. This would additionally need a whitelist check against the configured domain(s) for security reasons. This is not achievable with the current release of opentaxii, but would lead to the desired result with only 1 instance of opentaxii running.

@ropetin can you confirm that this is what you meant or clarify the question if it is not? Assuming it is, can you additionally comment on the possible solutions?

EDIT: Example response xml from a testrun to clarify what I meant in the first line. I'm talking about the "Address" bits for each service.

<taxii_11:Discovery_Response xmlns:taxii="http://taxii.mitre.org/messages/taxii_xml_binding-1" xmlns:taxii_11="http://taxii.mitre.org/messages/taxii_xml_binding-1.1" xmlns:tdq="http://taxii.mitre.org/query/taxii_default_query-1" message_id="2367565854442735220" in_response_to="123">
  <taxii_11:Service_Instance service_type="INBOX" service_version="urn:taxii.mitre.org:services:1.1" available="true">
    <taxii_11:Protocol_Binding>urn:taxii.mitre.org:protocol:http:1.0</taxii_11:Protocol_Binding>
    <taxii_11:Address>http://www.some-example.local/relative/path</taxii_11:Address>
    <taxii_11:Message_Binding>urn:taxii.mitre.org:message:xml:1.0</taxii_11:Message_Binding>
    <taxii_11:Message_Binding>urn:taxii.mitre.org:message:xml:1.1</taxii_11:Message_Binding>
    <taxii_11:Message>inboxA description</taxii_11:Message>
  </taxii_11:Service_Instance>
  <taxii_11:Service_Instance service_type="INBOX" service_version="urn:taxii.mitre.org:services:1.1" available="true">
    <taxii_11:Protocol_Binding>urn:taxii.mitre.org:protocol:https:1.0</taxii_11:Protocol_Binding>
    <taxii_11:Address>https://www.some-example.local/relative/path</taxii_11:Address>
    <taxii_11:Message_Binding>urn:taxii.mitre.org:message:xml:1.0</taxii_11:Message_Binding>
    <taxii_11:Message_Binding>urn:taxii.mitre.org:message:xml:1.1</taxii_11:Message_Binding>
    <taxii_11:Message>inboxA description</taxii_11:Message>
  </taxii_11:Service_Instance>
  <taxii_11:Service_Instance service_type="DISCOVERY" service_version="urn:taxii.mitre.org:services:1.1" available="true">
    <taxii_11:Protocol_Binding>urn:taxii.mitre.org:protocol:http:1.0</taxii_11:Protocol_Binding>
    <taxii_11:Address>http://www.some-example.local/relative/discovery</taxii_11:Address>
    <taxii_11:Message_Binding>urn:taxii.mitre.org:message:xml:1.0</taxii_11:Message_Binding>
    <taxii_11:Message_Binding>urn:taxii.mitre.org:message:xml:1.1</taxii_11:Message_Binding>
    <taxii_11:Message>discoveryA description</taxii_11:Message>
  </taxii_11:Service_Instance>
  <taxii_11:Service_Instance service_type="DISCOVERY" service_version="urn:taxii.mitre.org:services:1.1" available="false">
    <taxii_11:Protocol_Binding>urn:taxii.mitre.org:protocol:http:1.0</taxii_11:Protocol_Binding>
    <taxii_11:Address>http://www.some-example.local/relative/discovery-b</taxii_11:Address>
    <taxii_11:Message_Binding>urn:taxii.mitre.org:message:xml:1.0</taxii_11:Message_Binding>
    <taxii_11:Message_Binding>urn:taxii.mitre.org:message:xml:1.1</taxii_11:Message_Binding>
    <taxii_11:Message>discoveryA description</taxii_11:Message>
  </taxii_11:Service_Instance>
</taxii_11:Discovery_Response>

ropetin commented 2 years ago

@ropetin can you confirm that this is what you meant or clarify the question if it is not? Assuming it is, can you additionally comment on the possible solutions?

This is exactly what I'm talking about, yes. Option one is one of the things I tried, but I could not get it to work as expected. I tried three backends; SQLite3, MariaDB, and PostgreSQL, all with the same result: only one instance could successfully use the data at any time. I.e. I'd spin up the first instance, get some data in and everything would work like a champ. I'd then spin up the second instance, connect it to the backend and that would work fine, while the first instance started failing. Restarting the first instance would get that working again, but the second would now fail. Rinse and repeat. I don't have the error specifics because I destroyed that infrastructure, but can spin it back up if it would be helpful.

Option two definitely feels like the best solution and how I think most users would expect it to work. I don't believe this can be achieved outside of OpenTAXII (say with HAProxy) because it would require rewriting actual payloads.

erwin-eiq commented 2 years ago

I agree that option 2 is the best solution. It's not currently possible however. We need code changes that do the following things:

Accept a list of domains in the config
Check hostname of request against list, fail if it's not in the list (whitelist approach)
Use hostname of request when generating response

Let's turn this issue into a feature request for that.

I'm currently busy with adding taxii2 support, which has higher priority than this feature request so even though this is definitely a feature I'm willing to support I can't work on it right now. However, I'll gladly review and potentially accept a PR if you have time sooner than I do.

eclecticiq / OpenTAXII

Add multi-domain support #191