WIPACrepo / iceprod

IceCube dataset management system
MIT License
4 stars 4 forks source link

remove internal RPC in the server #124

Closed dsschult closed 7 years ago

dsschult commented 8 years ago

@ddelventhal asked a long time ago why we had modules in separate processes with IPC connecting them. At the time, this was necessary for throughput (keep modules in separate processes because of the GIL). I believe with the advent of futures and asynchronous python programming, this problem can now be completed without the internal RPC and separation of modules.

See https://github.com/dsschult/file_catalog for an example of running tornado + executor pool.

Strategy:

Advantages:

Disadvantages:

dsschult commented 8 years ago

Changing strategy slightly:

dsschult commented 8 years ago

diff stat:

 35 files changed, 2581 insertions(+), 5877 deletions(-)

And that's without touching the tests yet.

dsschult commented 7 years ago

groups branch is now merged:

83 files changed, 7224 insertions(+), 13275 deletions(-)