Piping across the network

[jacobbridges]

Piping data streams between machines would be an awesome feature. For instance tailing log files in several server instances and aggregating them in a centralized server via named pipes would be simple with cutie.

Not sure how to implement that though. A central server for handling cutie connections would probably be the easiest implementation. Also the server could cache some data so if the receiving pipes were down data loss would be minimized.

I think this should be available via an option, for example -g would signal a "global" pipe. For example:

# A normal cutie pipe
cat file | cutie

# A global cutie pipe
cat file | cutie -g

[Blastus]

QTpipe currently supports piping data over a network. It is as simple as providing the name or IP address of the computer that is acting as server. Would you like help merging such a feature into your code?

[jacobbridges]

qtpipe is currently throwing an error on my machine (OS X)

# What I tried
# In a terminal session:
$ cat data.txt | python qtpipe.py

# In another terminal session
$ python qtpipe.py > data_out.txt
Traceback (most recent call last):
  File "qtpipe.py", line 152, in <module>
    main()
  File "qtpipe.py", line 40, in main
    link.shutdown(socket.SHUT_RDWR)
OSError: [Errno 57] Socket is not connected

What methods would you recommend to resolve this issue?

---

Also as a footnote, how would you design the application to handle multiple "output" pipes on multiple machines? See issue #2 for more details.

[Blastus]

Have you tried commenting out line 40? What I really want to know is how the code is even getting that far without shutting down. Did any of the data get transmitted? It may be that the shutdown command is being sent from one socket to the other, and the other is getting automatically shut down on OS X. On Windows, the command is explicit on both instances. It might be better for the line 40 to be moved to just below line 36. That way, the sender ... just thought of something ... can send the shutdown command.

The idea that just came to mind is that technically there should be four shutdown commands -- one each before and after lines 34 and 36. A sender can say that there is nothing to read, and a receiver can say there is nothing to write. That would be more correct with how the code is being used. The needed changes have been pushed.

As for compatibility between Python 2 and 3, I am not aware of any version-specific code that exists. You can remove the first line or change it to test as needed. However, I would definitely recommend installing Python 3 on any machines that this is supposed to run on. The newest version of the language has far better differentiation when it comes to strings, bytes, and unicode. As written, strings are used for error messages, but all I/O in the program is handled using bytes.

[jacobbridges]

The data from data.txt is copied to data_out.txt before the error occurs. Not sure if it helps but I was using Python 3.4.3 to run your script.

Last night I was in a hurry so I didn't extensively test or modify your code. I just tried a simple cat and noticed the error. Does your program support tailing a file? cutiepipe continuously flushes any new data it receives to the stdout so if a stream (such as a tail) is broken the data will still be transferred.

I removed any reference to the difference between Python versions from my previous comment, as I realized the issue was opinion based. It would be very easy to maintain Python 2 and 3 on separate branches in this repository.

I'm currently working a rewrite of this project where the design is much more object-oriented. That code will be pushed to a "develop" branch soon and I would love to incorporate some of your changes into that code.