ivan-zapreev / Distributed-Translation-Infrastructure

The distributed statistical machine translation infrastructure consisting of load balancing, text pre/post-processing and translation services. Written in C++ 11 and utilises multicore CPUs by employing multi-threading, allows for secure SSL/TLS communications.
GNU General Public License v2.0
12 stars 3 forks source link

Unsuitable for running as a service #13

Closed jamesdbaker closed 4 years ago

jamesdbaker commented 6 years ago

When trying to run the servers as a service, the logs quickly get filled up with > symbols, which I assume is from the input prompt. An example script for creating and starting a service demonstrating this (on Ubuntu):

cat > /etc/systemd/system/mt-processor.service <<EOL
[Unit]
Description=MT Processor

[Service]
User=ec2-user
ExecStart=/opt/bin/bpbd-processor -c /opt/conf/processor.cfg
WorkingDirectory=/home/ec2-user

[Install]
WantedBy=multi-user.target
EOL

# Install and start service

sudo systemctl daemon-reload
sudo systemctl enable mt-processor
sudo systemctl start mt-processor

After a week or so, my disk is full up and when looking at what is taking up all the space I find it is /var/log/messages which is full of lines such as:

May 31 12:53:26 ip-10-2-3-171 bpbd-processor: >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>

Is it possible to modify so that it is suitable for running as a service? This happens for all three types of server.

ivan-zapreev commented 6 years ago

Dear James,

This must be happening with all the servers are they have the same server console implementation. However, this seems to be strange as the >> is printed only on input. So it is like you service is being constantly given some input. is this normal for when it is run as a service?

Kind regards,

Ivan

jamesdbaker commented 6 years ago

Yes, it's happening on all the servers.

I'm not sure whether it is normal or not, it doesn't sound like it should be but perhaps it's not supposed to work with applications that expect input.

ivan-zapreev commented 6 years ago

Just delivered something that might fix the issue, could you try out the latest push? I can not really reproduce the issue locally with ease ...

ivan-zapreev commented 6 years ago

I shall close the issue for now, please re-open if it persists

jamesdbaker commented 6 years ago

Thanks, I'll try it out at the start of next week when I'm next in the office.

ivan-zapreev commented 6 years ago

ok. great!

On Thu, May 31, 2018 at 6:21 PM, jamesdbaker notifications@github.com wrote:

Thanks, I'll try it out at the start of next week when I'm next in the office.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/ivan-zapreev/Distributed-Translation-Infrastructure/issues/13#issuecomment-393589401, or mute the thread https://github.com/notifications/unsubscribe-auth/AKH_HQazl9vo1yM-WJO8wDsbscZlhYHcks5t4BiKgaJpZM4UVAIp .

-- Best regards,

Dr. Ivan S. Zapreev http://www.tainichok.ru/

jamesdbaker commented 6 years ago

Apologies for the delay, but I've now tried this with release 1.8.1 and there is still an issue.

ivan-zapreev commented 6 years ago

Dear James,

Sorry to hear that :( So the quick fix did not work out, it will then require more time for investigating and fixing the issue. I recon some 1-2 days of work. Depends also on the type of solution.

Kind regards,

Ivan

ivan-zapreev commented 6 years ago

A valid work around is to run the infrastructure servers not directly as services but wrapped within screen command. Run under screen will allow them to be run as services and at the same time to still have access to the command-line console thereof.

ivan-zapreev commented 6 years ago

Dear James,

Could you please confirm if running under screen is a workable workaround for you?

Kind regards,

Ivan

jamesdbaker commented 6 years ago

Hi Ivan, yes I believe that will work ok. James