Inter-server Communication

[th0mcumm1ngs]

Feature Purpose: There needs to be a system devised for communication between the servers.

The Original Idea

The original idea for this, with early versions, was that each server would have two separate sender.py and receiver.py files that would contain HTTP request sending actions and a flask server respectively. Each server would then send JSON packages, which would each contain a standardized data structure containing an action, its data, etc.

This was an okay approach as a first draft. However, there are much more efficient, already-existing protocols and methods that specialise in this field, rather than 'reinventing the wheel' using web requests.

Newer Ideas

The most up-to-date idea – more research will be conducted before development of this feature – is similar to the first in that data will be in JSON packages, using a standardised format. However, instead of HTTP requests using verbose files and messy programming, the server will use Sockets and a request queue of some sort that the main.py file will fetch from based on an instruction scheduling system (FCFS to start with, probably more optimised / prioritised ones in the future).

Each server will hopefully then be able to send to another's queue and work through its own queue simultaneously, requiring no more than one python file when working asynchronously.

[th0mcumm1ngs]

I did some research today, here's how it's going to work:

• Each server will have Socket send and receive methods, using an asynchronous server and client.
• These will send JSON data which is encrypted by a single secret key that each server has access to (the security of this will be greatly increased once it's up and running).
• The data will then be stored in a message queue. I'm looking into having a single JSON file for each server, but this could cause conflicts due to it being written to by multiple files at once. There must be a fix for this.

[th0mcumm1ngs]

I think that a local SQLite queue might be better

[th0mcumm1ngs]

Plan of Action

Note: I'm taking a trial and error approach to this as I think it will be better than trying to plan, which always takes too long and isn't helpful enough. This is like a rough sketch model.

• [ ] Create a module that contains all the transmission procedures (socket client and server), which will all be able to run asynchronously or in a separate thread (both?[^2]), and put them on each server.
• [ ] Import modules to main.py file of each server.
• [ ] Add secure encryption for every data transfer.[^1]
• [ ] Add a hook in the server part so that a task queue can be added in a later feature

[^1]: Data should only be decrypted when it needs to be read, so any data stored in SQL databases or anywhere else will be encrypted. The code can then decrypt (and re-encrypt) when required. [^2]: The code could have the server running in a separate thread to catch incoming communications, while each client's action could be asynchronous in the main thread to avoid slowing down the program.