Design and implement a better progress reporting system for multiprocessing builds.
The system should permit a remote or sub process to record it's current operation, and maybe it's debug log. The system can be specific to Postgres, since multiprocessing doesn't work with Sqlite.
There should be a new progress logging object, either global or attached to the library, that saves log records to the database. Each record includes the host name and pid of the process. THe records include information about:
bundle ( vid ) for current operation
text of current operation
progress rate ( records / second )
progress completion ( # records completed, # total )
Text and stack trace of exceptions.
Job Queue
Alternatively, this system could be a more general job queue, allowing for jobs to be distributed. In that case, the parent process would create multiple job records, and other processes would get information about the jobs from the database, then write progress back to the database.
Design and implement a better progress reporting system for multiprocessing builds.
The system should permit a remote or sub process to record it's current operation, and maybe it's debug log. The system can be specific to Postgres, since multiprocessing doesn't work with Sqlite.
There should be a new progress logging object, either global or attached to the library, that saves log records to the database. Each record includes the host name and pid of the process. THe records include information about:
Job Queue
Alternatively, this system could be a more general job queue, allowing for jobs to be distributed. In that case, the parent process would create multiple job records, and other processes would get information about the jobs from the database, then write progress back to the database.