Ability to write images to stdout or pipe in command-line mode

[adolski]

It would be great if RawTherapee were able to write images to stdout in command-line mode. This would be super useful for invoking from other tools and capturing the output in memory.

I have tried doing something like:

rawtherapee -o /dev/stdout -t -Y -c image.dng

In this case, RT tries to create a file called /dev/stdout.tif.

If I try to get clever and do:

ln -s /dev/stdout stdout.tif rawtherapee -o stdout.tif -t -Y -c image.dng

This actually almost works, but console messages appear in the output, causing it to be corrupt.

So maybe what is needed is either a -q (quiet) flag, or, better, to make the -o flag accept a special token to signify stdout, which would also suppress console messages. (That would work on Windows too.)

[Benitoite]

Maybe console status messages could be bifurcated over to an error stream stderr for image data to own stdout.

[Waveluke]

I'd definitely find this very useful, as for doing raw processing on high resolution raw video, it's easy to quickly run out of hard drive space with intermediary tiffs before encoding with ffmpeg. 4k 3:1 dng files are 3.5 megabytes a pop, but the tiffs are 48 megabytes uncompressed, and only about possibly 36-44 compressed, so the files quickly fill up hard drives, even when the original raw files, and subsequent 10 bit Prores produced by ffmpeg is totally manageable. This effectively caps max clip length at 20 minutes of 4k per terabyte of free space unless I want to write some convoluted segment encoding script, not to mention all the unnecessary slowdown from hard drive IO.

[Thanatomanic]

@Benitoite Do you see any problem redirecting all console logging/verbose output to stderr? Because that should be quite easy to do.

[Thanatomanic]

@Waveluke I've quickly cooked up a branch silence-output that should forward all intermediate output of RT to stderr instead of stdout. Could you give this a test for your setup?

[heckflosse]

@Thanatomanic What about the cout ?

[Thanatomanic]

Isn't cout the C++ equivalent of stdout? If I understand things correctly, it is particularly this stream that should remain 'clean', no?

[Floessie]

@Thanatomanic Sure, but there are places using std::cout which also need to be changed.

[Thanatomanic]

I've pushed a change in the branch to include std::cout. That is all redirected to std::cerr now.

[Thanatomanic]

So far I have not touched main.cc and main-cli.cc since they are likely not interfering and also provide useful 'normal' output.

[heckflosse]

Hmm, imho this one https://github.com/Beep6581/RawTherapee/blob/dev/rtgui/main-cli.cc#L216 should also be std::cerr

[Thanatomanic]

@heckflosse Thanks, I've actually gone through all cout's in main-cli.cc and changed most of them to cerr. The only ones still cout are those for the text when there is no input provided to the program.

@Waveluke I'm really curious if you can manage to get your pipeline to work this way. Take your time and please provide feedback.

[Waveluke]

OK, I am a bit noobish on this TBH, (also mistakenly checking PIXLS more than here (whoops)). Would the latest auto-compiled dev version have this change?

Edit: I found that there is a separate silence output branch that you guys recently did. This would be my first time compiling Rawtherapee from scratch. How would I know if something doesn't work, it being something on my end (not compiling correctly) vs. your push not solving the issue?

Running Windows 10.

[Beep6581]

I've quickly cooked up a branch silence-output that should forward all intermediate output of RT to stderr instead of stdout. Could you give this a test for your setup?

Hey @Thanatomanic , is it correct that branch silence-output has not been merged into dev yet?

Is there now a way to pipe an image from rawtherapee-cli either in dev or in silence-output?

[wuziq]

silence-output branch with stdout symlink?

I tried the silence-output branch along with OP's suggestion of symlinking to stdout. I could be doing something wrong but it doesn't appear to be working:

wuziq@wuziq-hp:~/programs/rawtherapee$ ln -s /dev/stdout tifdump.tif
wuziq@wuziq-hp:~/programs/rawtherapee$ ./rawtherapee-cli -o tifdump.tif -q -s -t -b8 -Y -c /mnt/d/images/DSC04397.ARW
RawTherapee, version 5.8-3014-g94b5f7232, command line.
Output is 8-bit integer.
Processing: /mnt/d/images/DSC04397.ARW
  Merging sidecar procparams.
II*
v

~
"
'
d!2
d0230












bU  



|n   F@

Z


E

 
bU

 


ur
     
W

 246
#
2021:07:17 18:51:542021:07:17 18:51:54A








FE 20mm F1.8 G  

 
 !
"
#
$
%
&
)







 
 
 
 
 
'
A

cI
J
K
O
P
R
tStandard
%pppŁ
戀 
     aJ0
Rgc4tqw_p
IJJJJ
JpŁ
}n

^p}ppppyppDpp*ppW^ii"iW  00`RR(+p}ppppyppDpp*ppW^ii"iWx{ 00Ծ؁l(lp}ppppyppDpp*ppW^ii"iWx{ 00Ծ؁l(lp}ppppyppDpp*ppW^ii"iWx{ 00Ծ؁}(RJl



111QQQQQQQQQQQQQQQQ}
p

Ł


@
J@0tiV^]`Q


(m:ԙ>'qp}
9
H
ppGh@m;

tjH
߬p





p

JĜ

pppeA4
''

JJl
l.}[}[}[}[}"/}VJJ'JJ$@ĕJ
C(EAQ
8q$


+
+
]\$]\
^3333}}}}?4A]_0@>u.
k*~}
Ec
%]smsx!gfݝTIFFAppendToStrip: Maximum TIFF file size exceeded.
TIFFWriteDirectoryTagData: IO error writing tag data.
^C

Update RT to stream output?

I wanted to see whether I could update RT to stream its output, namely TIFFs, directly to stdout. But, unless I'm missing something, it doesn't seem straightforward to stream TIFFs to stdout since RT depends on libtiff for writing files, and TIFFOpen() just takes a const char *. I'm considering adding the option to libtiff.

Using named pipes?

So I tried using a named pipe, but I don't think libtiff likes it, though I'm likely doing something wrong. I wanted to try piping one TIFF into ffmpeg. In one console:

wuziq@wuziq-hp:~/programs/rawtherapee$ mkfifo tifdump.tif
wuziq@wuziq-hp:~/programs/rawtherapee$ file tifdump.tif
tifdump.tif: fifo (named pipe)

In another console:

wuziq@wuziq-hp:~/programs/rawtherapee$ exec 3<>tifdump.tif
wuziq@wuziq-hp:~/programs/rawtherapee$ ffmpeg -r 24 -f image2pipe -i pipe:3 -c:v libx264 -preset slow -profile:v high -crf 18 -pix_fmt yuv420p delete.mp4
ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)
  configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[ffmpeg waits for input here]

Back in the first console:

wuziq@wuziq-hp:~/programs/rawtherapee$ ./rawtherapee-cli -o tifdump.tif -q -s -t -b8 -Y -c /mnt/d/images/DSC04397.ARW
RawTherapee, version 5.8-3014-g94b5f7232, command line.
Output is 8-bit integer.
Processing: /mnt/d/images/DSC04397.ARW
  Merging sidecar procparams.
TIFFAppendToStrip: Maximum TIFF file size exceeded.
TIFFWriteDirectoryTagData: IO error writing tag data.
[RT is stuck here]

ffmpeg comes alive in the second console:

[image2pipe @ 0x562612dbe580] Could not find codec parameters for stream 0 (Video: tiff, none): unspecified size
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Input #0, image2pipe, from 'pipe:3':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: tiff, none, 24 tbr, 24 tbn, 24 tbc
Output #0, mp4, to 'delete.mp4':
Output file #0 does not contain any stream

Finally, if I don't try reading from pipe at all:

wuziq@wuziq-hp:~/programs/rawtherapee$ ./rawtherapee-cli -o tifdump.tif -q -s -t -b8 -Y -c /mnt/d/images/DSC04397.ARW
RawTherapee, version 5.8-3014-g94b5f7232, command line.
Output is 8-bit integer.
Processing: /mnt/d/images/DSC04397.ARW
  Merging sidecar procparams.
TIFFAppendToStrip: Maximum TIFF file size exceeded.
TIFFWriteDirectoryTagData: IO error writing tag data.
TIFFWriteDirectoryTagData: IO error writing tag data.
Error saving to: tifdump.tif

Is there anything wrong with what I'm doing above?