The cdnservers store the video access information in the nginx logfile. These files are rotated daily (see #47) and then should be transferred to the central cdnmanager server.
The cdnmanager can do it using a daily celery task, that logs into the cdnserver and retrieves the log file. The logfile has a unique name (the hostname and the date are included in the name) so you can just put it into a single directory for now.
Notes:
In the future we may need to keep the files in separate directories to avoid having thousands of files there, which make administration hard, but that's not a problem for now.
The servers IP addresses are stored in the database. We can use a single task that retrieves the files over all servers, or we can create a separate task for each server. (example: the daily task loops over the servers and dispatches the actual transfer tasks, which makes for shorter tasks, instead of a big one).
There is already some code to distribute files to the servers using paramiko.sftp that can be used as a reference to retrieve the logfiles.
The cdnservers store the video access information in the nginx logfile. These files are rotated daily (see #47) and then should be transferred to the central cdnmanager server.
The cdnmanager can do it using a daily celery task, that logs into the cdnserver and retrieves the log file. The logfile has a unique name (the hostname and the date are included in the name) so you can just put it into a single directory for now.
Notes: