search

tidymodels / planning

Documents to plan and discuss future development
MIT License
37 stars 4 forks source link

Feature engineering for image data #17

Open gustavomodelli opened 3 years ago

gustavomodelli commented 3 years ago

Image Classification with Tidymodels

Suggest Recipe: step_image

Is It possible to create a recipe that makes image processing?

Is It possible to process images like array reshape in Keras?

The tidymodels are an excellent package but there are no options to image process

topepo commented 3 years ago

It's a great and would really improve the modeling environment.

I think that it would require the images to be stored within the recipe in a different format that is more conducive to how the data are usually handled. This could get a little tricky but @EmilHvitfeldt's textrecipes does something very similar for text and it works very well.

If you are interested in starting a side package to do this, let us know. Otherwise, this might be lower down the line in priority unless @dfalbel or @skeydan are interested. They have a better handle on processing tools for those data.

EmilHvitfeldt commented 3 years ago

sounds like an interesting idea. I haven't done a lot of work with images myself. the main issue you might run into is that recipes only allows output to be 2-dimensional and I don't know if that will be troublesome.

If you need to talk to someone who has spent some time handling non-rectangular data inside recipe then I'm happy to talk.

dfalbel commented 3 years ago

In theory it would be possible to use {tfhub} for that.

We have implemented steps for text embedding here: https://github.com/rstudio/tfhub/blob/master/R/recipe.R and we could have similar functionality for images.

I imagine that images would need to be stored in an list-column where each element is an array.

juliasilge commented 3 years ago

I'm moving this to our planning repo since this is more about long term ideas (and probably a separate package) than a feature request here in recipes. 👍