search

TIBCOSoftware / catalystml

CatalystML is an open source specification for real-time feature processing, purpose built to transform data for machine learning models.
BSD 3-Clause "New" or "Revised" License
2 stars 3 forks source link

include a specific type for Images? #8

Closed abramvandergeest closed 5 years ago

abramvandergeest commented 5 years ago

Should we have a specific type just for images? We could just have it as byte arrays, but so much more is in an image type (headers etc.)

@mellistibco @fm-tibco @skothari-tibco @danrope @jagannathyv

abramvandergeest commented 5 years ago

So @fm-tibco and I talked about this during a meeting today. We thought of 3 possible types to add that could address this issue:

1)bytes (and hence bytes array) 2) file 3) image

We decided that bytes isn't really (normally) in the Transform part of ETL and is more regularly in the Extract part. We might want to add it later, but it isn't part of the core and that handling images needs to be part of the core.

For file we decided that would complicate IO (which we thought should be independent of CML) and make it hard to deploy in containers or a serverless.

This led to us thinking that a Image type was probably the way to go that was most language independent. The Golang implementation would probably have byte arrays in the back end, but that is an implementation choice.

abramvandergeest commented 5 years ago

@mellistibco @fm-tibco and I decided that the image type was the correct way to go and it will be a byte array within the golang implementation.