Support for stdio and piping

[alterae]

It would be handy if the Typst CLI supported input via stdin, and perhaps also output via stdout.

Usage example:

$ command-that-outputs-typst-markup | typst > processed.pdf

[johannes-wolf]

Outputting to stdout already works by passing /dev/stdout as output file.

[Dherse]

Arguably that only works on Linux/macOS, and you should expect most users to be either on Windows or on the web

[owiecc]

Doesn't Windows have pipes? Not sure how the web ties into this. Most web servers run on unix systems.

[Dherse]

Doesn't Windows have pipes?

Windows does have pipes, I was answering to another comment suggesting passing /dev/stdout as the output file would work. Which is not something that you can do on Windows, however, having a --stdout could be done.

Not sure how the web ties into this. Most web servers run on unix systems.

The web system runs in WASM so there would be no stdout there anyway, this would be a CLI-only feature if it were to be implemented.

[pepa65]

It would be great to be able to pipe/redirect into typst (preferably with the - syntax, like: echo Test |typst - out.pdf). Then you don't need to create an actual/virtual file to provide input.

This also doesn't work: typst c <(echo Test) out.pdf: error: input file not found (searched at /dev/fd/63).

This should normally work in Rust, I don't understand why it wouldn't...

[edgarogh]

I kinda implemented a fix. It's not super clean, conservatively only enables this special case on Unix, and doesn't care to check if /dev/stdin actually exists. If I can open a PR and/or fix a few things so it gets merged upstream, I'd gladly do so :wink:. For the time being, it suits my own needs.

[Enter-tainer]

Can you give a few use case when this is useful? cc @s-cerevisiae I feel it is somehow strange because:

1. typst source file can import files in the same directory. what should we do if the file from stdin imports other files?
2. per #2894 we can already pass arbitrary data to cli using args. Do we still need accept input from stdin?

[Enter-tainer]

One thing I can come up with is that when integrating typst-cli to other systems. If a) you don't want touch filesystem b) you can embed everything into a single typ file, you can use piping: sending data into stdin and receive from stdout.

[pepa65]

What you would expect to normally work on Posix-type systems doesn't somehow. It enhances scriptability, it conforms to normal expectations. I don't know about Windows, but for all the rest this should be true.

[s-cerevisiae]

@Enter-tainer I'm integrating typst with my Zola setup to render inline formulas and do complicated layouts. It involves hundreds of different, tiny inputs so I needed to create and delete that many temporary files before this. People other than me have also encountered this. It will also come in handy when the file needs pre-processing, although that's not my use case.

For the two questions:

1. The current implementation treats current working directory as the project base to resolve relative imports, which is already the default when typst can't determine one.
2. Yes, afaik it's treated as data and not code. I don't want to abuse eval for that, which is strictly worse. (e.g. the error message will be unreadable)

[C0ffeeCode]

Is there someone working on this? I still have work on piping support what I thought to include in #2894 but then did not.

[istudyatuni]

I guess this can be closed now? @laurmaedje

[laurmaedje]

Yep! Thanks for the heads-up.

[pepa65]

This now works: echo Hallo |typst c - out.pdf This does not: typst c <(echo Hallo) out.pdf

[C0ffeeCode]

Is it now possible to pipe data not as an input document but as an input variable? For example, like typst compile --input key_of_piped_data=[...] whereas [...] is the data from the pipe.

[laurmaedje]

@pepa65

This does not: typst c <(echo Hallo) out.pdf

It works on my machine (macOS). Could you clarify what exactly doesn't work?

@C0ffeeCode How exactly it works depends on your shell. In bash you can write:

typst compile file.typ --input val=$(echo "Hello")

[Apollo314]

Is it now possible to pipe data not as an input document but as an input variable? For example, like typst compile --input key_of_piped_data=[...] whereas [...] is the data from the pipe.

you could do

typst compile --input key_of_piped_data="$(cat)" file.typ

[jakobjpeters]

The linked PR only implemented reading from stdin, is there a problem with or reason not to write to stdout?

[laurmaedje]

Stdout was also implemented, but in another PR: https://github.com/typst/typst/pull/3632

[jakobjpeters]

Stdout was also implemented, but in another PR: #3632

Awesome, thank you!

[pepa65]

@laurmaedje

Could you clarify what exactly doesn't work?

typst c <(echo Hallo) out.pdf

error: input file not found (searched at /dev/fd/63) (typst 0.11.1)

[laurmaedje]

This now works: echo Hallo | typst c - out.pdf This does not: typst c <(echo Hallo) out.pdf

Okay, yeah. I can reproduce it in Docker, so for some reason it fails on Linux but works on macOS. I'm not that knowledgeable w.r.t. piping on Linux, but I'll share this in #contributors on Discord. Maybe someone has an idea.

[alerque]

The second form creates a temporary node that is a named pipe. Typst is handling input on STDIN as a stream, but it is trying to treat the file name given for the named pipe as a regular file, which is not possible (it is not seekable, it actually a stream because the app producing it (in this simple case echo) doesn't start doing any work until something starts reading from the pipe). An extra check is going to be required to handle named pipes vs. regular files. It looks like it's getting tripped up before even trying to open the file just trying to canonicalize the name.