gnembon / fabric-carpet

Fabric Carpet
MIT License
1.72k stars 275 forks source link

[Scarpet] Support for reading Image files as an array of pixels #599

Open replaceitem opened 3 years ago

replaceitem commented 3 years ago

See a relevant discord discussion here. This could be usefull for blockart stuff or similar. I will probably make a PR on this soon. @gnembon what about image writing? could be used to export maps of a server or stuff like that?? i could do both, if you feel like its a good addition to scarpet

Ghoulboy78 commented 3 years ago

btw how will you do that? will you store as 2d array of NumericValues?

replaceitem commented 3 years ago

Not quite sure yet how it would be the best to do (in terms of memory and cpu usage) Either being able to read individual pixels form a file (though that would mean the file would have to be opened and closed for each read, but that would save memory and not use a huge list) Or returning the full list, saving some cpu but being huge in memory I'd love some opinions and advice on how that should be implemented

gnembon commented 3 years ago

well, I doubt you would be to run into memory issues with the usecases for pixelarts etc, where the images would be low resolution anyways. And better memory optimized access via own methods (potentially defining its own container type would be clunky to implement. For these reasons I would lean towards image:x:y kinda access model (or :y:z, I am not discriminating against your coordinate preferences). But, (and that's a big but), defining a new n-dimensional array of float types (tensors) (I would stick with doubles for practical reason, despite the industry swaying towards float32, float16 or even less to save memory and compute), as an efficient way of representing tensors, is something that I would be glad to see, potentially enabling neural networks and automatic inference in scarpet without being very inefficient. All of this while handling images as a sideeffect.

Another question is pixel color representation - would that be NxNx3(4) where each color has its own value, or NxNx1 where each value uses already existing standard for passing color values in shapes of 0xRRGGBBAA (as an int, perfectly contained in any double value). In any case a relevant methods to decode colors would be very welcomed (color2rgba and rgba2color) - could be done as a function, but that would be slower and useful anyways.

Obviously saving would work as well. In that case read_file and write_file would operate / require tensor types. Suggesting using more universal name tensor vs more specific to one implementation ndarray or something similar

gnembon commented 3 years ago

Actually if this gets implemented with support for tensors (started liking that even more), that would cover #593 partially, which is a very old suggestion.

Ghoulboy78 commented 3 years ago

ooh, so you could have a dynamic way of representing the layers of a neural network! this looks cool!

Ghoulboy78 commented 3 years ago

In any case a relevant methods to decode colors would be very welcomed (color2rgba and rgba2color) - could be done as a function, but that would be slower and useful anyways.

lol hypocrisy #374