search

gulpjs / vinyl

Virtual file format.
MIT License
1.28k stars 105 forks source link

Remove streaming content support? #133

Open phated opened 7 years ago

phated commented 7 years ago

Streaming contents have made many things hard. I have some ideas about ways to replace some of the use cases and need to explore more to see if we can remove streaming contents completely.

Marak commented 7 years ago

I'm a bit confused by this issue. If we remove streaming contents completely, wouldn't that break much of the vinyl / gulp ecosystem?

Also, if we remove streaming, wouldn't that mean that all files are going to be buffered into memory, or is that a vinyl-fs responsibility?

Any clarifications would be appreciated. I'm probably misunderstanding the issue.

phated commented 7 years ago

@Marak this change would be a breaking change to the current Vinyl spec where the contents property can be a Buffer, Stream or null. However, it likely wouldn't break much of the gulp ecosystem because most plugins don't support streaming. The hope behind streaming contents was that more and more modules would support streaming over time (a good example being uglifyjs); however, over time, that hope has not held true and so plugins need to check if contents are streaming and most just error out.

This issue was really inspired by your desire to have a toJSON method, which cannot be supported if we allow for streaming contents and further compounded by the complexities of putting streams within streams (vinyl-fs, etc). I think we can come up with better solutions but it hasn't been explored yet.

This won't come until after gulp 4 which will include vinyl-fs 3.x/vinyl 2.x

Marak commented 7 years ago

I agree that correctly implementing streaming modules for gulp eco-system is not super easy. A lot of the plugins I've seen don't bother to stream at all.

I'm still doing toJSON in my fork and it's working well. I've recently made some changes so fs.Stat properties are also serialized. Pretty much my use-case has been serializing vinyl instances to be sent over the wire. In some cases I don't care about contents when calling toJSON, I just need the metadata and stats, so I don't mind if I lose the stream or buffer reference.

Marak commented 7 years ago

RE: #83

phated commented 7 years ago

@Marak I'd love for better-stats to be properly serializable - node's fs.Stat leaves so much to be desired.

phated commented 5 years ago

Also need to review https://www.npmjs.com/package/smart-buffer