Closed mgendarme closed 5 years ago
Hi,
can you please provide the structure and content (by example) of X
and Y
?
Without seeing the model definition and the data, what I'd conclude from the error message is that given the amount of data is so small (48 if I'm correct?), and you do a validation split of 4:1, that sometimes the validation split does not have all (2 I guess?) classes.
By the way I don't see how the StackOverflow link addresses this problem? There, the setup is pretty different as they are streaming files from directories, not doing a random split on in-memory data.
Perhaps you could switch to the streaming method too? Then it would be pretty clear what's in the validation set.
Hi Sigrid, Thanks for the reply. I updated the post with many more details to give more context. If example images are required I can post that as well.
In the python example they did stream data from directory instead of the memory. I may be naively assumed that the difference between flow_images_from_directory and flow_images_from_data was just the location from which the data are fetched (disk VS memory).
Don't I generate new images on the fly while usingflow_images_from_data(X, Y, image_generator, ...)
?
Many thanks for your help.
Cheers
Thanks for the clarifications. Now that you say it's mainly about the data augmentation, I think we should not investigate that specific error message first (which originally lead me to say that probably you could avoid this using the flow-from-directory approach used in the SO example), but concentrate on the data augmentation...
I think that probably, the way you pass in the data the targets are not transformed (I think).
In the Keras docs, there is an example of data augmentation for segmentation - search for "Example of transforming images and masks together.".
Actually I'm copying it here for further reference:
# we create two instances with the same arguments
data_gen_args = dict(featurewise_center=True,
featurewise_std_normalization=True,
rotation_range=90,
width_shift_range=0.1,
height_shift_range=0.1,
zoom_range=0.2)
image_datagen = ImageDataGenerator(**data_gen_args)
mask_datagen = ImageDataGenerator(**data_gen_args)
# Provide the same seed and keyword arguments to the fit and flow methods
seed = 1
image_datagen.fit(images, augment=True, seed=seed)
mask_datagen.fit(masks, augment=True, seed=seed)
image_generator = image_datagen.flow_from_directory(
'data/images',
class_mode=None,
seed=seed)
mask_generator = mask_datagen.flow_from_directory(
'data/masks',
class_mode=None,
seed=seed)
# combine generators into one which yields image and masks
train_generator = zip(image_generator, mask_generator)
model.fit_generator(
train_generator,
steps_per_epoch=2000,
epochs=50)
I suggest porting this to R and seeing if that works correctly.
Thanks a lot Sigrid! I think I had seen that bit of information at some point. I tried it and here is how I think it should look like:
# we create two instances with the same arguments
# 1. Define your data augmentation
data_gen_args = list(rotation_range = 4,
horizontal_flip = TRUE,
vertical_flip = TRUE)
image_datagen = image_data_generator(data_gen_args)
mask_datagen = image_data_generator(data_gen_args)
# Provide the same seed and keyword arguments to the fit and flow methods
# 2. Fit the augmentation
seed = 1
image_datagen %>% fit_image_data_generator(X,
augment = T,
seed = seed) # seed seems to be the problem
mask_datagen %>% fit_image_data_generator(Y,
augment = T,
seed = seed) # seed seems to be the problem
# 3. Setup your generator using flow_from_directory()
image_generator = flow_images_from_data(X,
image_datagen,
seed = seed)
mask_generator = flow_images_from_data(Y,
mask_datagen,
seed = seed)
# combine generators into one which yields image and masks
train_generator = list(image_generator, mask_generator)
# 4. Train your model with fit_generator()
early_stopping <- callback_early_stopping(patience = 8)
model %>% fit_generator(train_generator,
steps_per_epoch = 16,
epochs = 32,
verbose = 1,
callbacks = list(early_stopping))
Here are the issues I am running into after running:
image_datagen %>% fit_image_data_generator(X,
augment = T,
seed = seed)
Rerun with Debug
Error in py_call_impl(callable, dots$args, dots$keywords) :
TypeError: Cannot cast array from dtype('float64') to dtype('int64') according to the rule 'safe'
I did search for that specific issue but could not come up with a solution.
As a side not I did find this as well and am starting to investigate it deeper. Could be another solution for the same problem. Though if this is the case it is a lot of work for a "simple" dual Image / Mask tandem augmentation task.
Cheers
Can you change seed <- 1
to seed <- 1L
?
This snippet works for me:
seed = 1L
X <- k_random_uniform(shape = c(10, 256, 256, 3)) %>% k_eval()
image_datagen %>% fit_image_data_generator(
X,
augment = T,
seed = seed
)
Thanks Sigrid that solved the first issue. Now it gets stuck when I run:
image_generator = flow_images_from_data(X,
image_datagen,
seed = seed)
Providing me with:
Error in dim(x) <- length(x) : invalid first argument
Though the dimensions of X did not change from fit_image_data_generator
to flow_image_from_data
.
The structure of X at this point looks like:
> str(X)
num [1:48, 1:512, 1:512, 1] 0.0839 0.0888 0.3018 0.2063 0.0786 ...
Cheers
Do you think you could post a complete chunk of code I could run to reproduce? Ideally I could just run that with random data like I did in my last comment? Btw does it fail in the same way if you use random data like I did?
I managed to solve the problem, it was pretty easy: This:
image_generator = flow_images_from_data(X,
image_datagen,
seed = seed)
should be turned into this:
image_generator = flow_images_from_data(X,
generator = image_datagen,
seed = seed)
The aim now is to feed the fit_generator function. This is what I have at the moment:
data_gen_args = list(rotation_range = 4,
horizontal_flip = TRUE,
vertical_flip = TRUE)
image_datagen = image_data_generator(data_gen_args)
mask_datagen = image_data_generator(data_gen_args)
# Provide the same seed and keyword arguments to the fit and flow methods
# 2. Fit the augmentation
seed = 1L
image_datagen %>% fit_image_data_generator(X,
augment = T,
seed = seed)
mask_datagen %>% fit_image_data_generator(Y,
augment = T,
seed = seed)
# 3 Setup generators using flow_from_directory() and combine them into one which yields image and masks
iter <- 0
train_generator = function(X,
Y,
total_iter){
Xaug <- X
Yaug <- Y
function(){
iter <<- iter + 1
if (iter <= total_iter) {
Xaug = flow_images_from_data(Xaug,
generator = image_datagen,
seed = seed)
Yaug = flow_images_from_data(Yaug,
generator = mask_datagen,
seed = seed)
result <- list(Xaug,
Yaug)
return(result)
} else
NULL
}
}
# for testing I set total_iter to 2
train_iterator = py_iterator(train_generator(X = X, Y = Y, total_iter = 2))
# 4. Train your model with fit_generator()
early_stopping <- callback_early_stopping(patience = 8)
model %>% fit_generator(train_iterator,
steps_per_epoch = 16,
epochs = 32,
verbose = 1,
callbacks = list(early_stopping))
My problem is at step 3 I guess. I do get that I have to provide an iterator function that should provide a list containing (inputs, targets). However when I feed my model with the train_iterator I have the feeling it is computing endlessly (or I do not get an error message telling that an something went wrong) despite the rule applied to the train_generator. I am obviously doing something wrong at step 3 but do not know what exactly. Any idea where the problem is suppose to be? Cheers
To me this looks like you have a generator (flow_images_from_data
) inside a generator (train_generator
).
How about trying the simpler approach from the Python documentation?
Hi Sigrid, Actually I am planning two steps ahead as I want to use a custom made image_generator that would not be limited by the features implemented right now (and also apply intensity based transformation to the image but not to the mask). I have seen that the problem I mentioned earlier was already reported somewhere else but not answered. I do get the same "freeze" as the person who wrote the post on stackoverflow. I also get this problem when trying to run this code with the original data from the kaggle chalenge. In case it helps here are my session info:
R version 3.5.1 (2018-07-02)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.1 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=de_DE.UTF-8 LC_COLLATE=en_US.UTF-8 LC_MONETARY=de_DE.UTF-8
[6] LC_MESSAGES=en_US.UTF-8 LC_PAPER=de_DE.UTF-8 LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=de_DE.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] parallel stats graphics grDevices utils datasets methods base
other attached packages:
[1] bindrcpp_0.2.2 doMC_1.3.5 iterators_1.0.10 foreach_1.4.4 reticulate_1.10 EBImage_4.24.0 forcats_0.3.0 stringr_1.3.1 dplyr_0.7.7
[10] purrr_0.2.5 readr_1.1.1 tidyr_0.8.2 tibble_1.4.2 ggplot2_3.1.0 tidyverse_1.2.1 keras_2.2.0.9001
loaded via a namespace (and not attached):
[1] Rcpp_1.0.0 locfit_1.5-9.1 lubridate_1.7.4 lattice_0.20-38 fftwtools_0.9-8 png_0.1-7 assertthat_0.2.0
[8] zeallot_0.1.0 digest_0.6.18 R6_2.3.0 tiff_0.1-5 cellranger_1.1.0 plyr_1.8.4 backports_1.1.2
[15] httr_1.3.1 pillar_1.3.0 tfruns_1.4 rlang_0.3.0.1 lazyeval_0.2.1 readxl_1.1.0 rstudioapi_0.8
[22] whisker_0.3-2 Matrix_1.2-15 htmlwidgets_1.3 RCurl_1.95-4.11 munsell_0.5.0 broom_0.5.0 compiler_3.5.1
[29] modelr_0.1.2 pkgconfig_2.0.2 BiocGenerics_0.28.0 base64enc_0.1-3 tensorflow_1.10 htmltools_0.3.6 tidyselect_0.2.5
[36] codetools_0.2-15 crayon_1.3.4 withr_2.1.2 bitops_1.0-6 grid_3.5.1 nlme_3.1-137 jsonlite_1.6
[43] gtable_0.2.0 magrittr_1.5 scales_1.0.0 cli_1.0.1 stringi_1.2.4 xml2_1.2.0 generics_0.0.1
[50] tools_3.5.1 glue_1.3.0 hms_0.4.2 jpeg_0.1-8 abind_1.4-5 yaml_2.2.0 colorspace_1.3-2
[57] rvest_0.3.2 bindr_0.1.1 haven_1.1.2
Any idea where this could be coming from?
Cheers
hi,
let's do this step by step then.
Regarding the code from SO, try removing the py_iterator
which is not need in current versions.
E.g. (based on that code)
t <- mikes.custom.generator.function()
t
play.network %>% fit_generator(
t,
steps_per_epoch = 1,
epochs = 2
)
Btw here
https://blogs.rstudio.com/tensorflow/posts/2018-11-05-naming-locating-objects/
are several examples of R generators (simple ones).
Hi Sigrid,
Thanks a lot for your help, removing the py_iterator did solve the problem. What I have now looks like this:
custom_generator <- function(data, # tibble containing encoded X and Y
shuffle,
batch_size) {
i <- 1
function() {
if(shuffle) {
indices <- sample(1:nrow(data), size = batch_size)
} else {
if (i + batch_size >= nrow(data) )
i <<- 1
indices <- c(i:min(i + batch_size - 1, nrow(data)))
i <<- i + length(indices)
}
custom_augmentation(data) # perform geometry based operation on X and Y and intensity based operations on X only
list(X,Y)
}
}
train_generator <- custom_generator(data = train_input,
shuffle = TRUE,
batch_size = BATCH_SIZE)
val_generator <- custom_generator(data = val_input,
shuffle = FALSE,
batch_size = BATCH_SIZE)
early_stopping <- callback_early_stopping(patience = 8)
history <- model %>%
fit_generator(
generator = train_generator,
epochs = 100L,
steps_per_epoch = as.integer(nrow(train_input) / BATCH_SIZE),
validation_data = val_generator,
validation_steps = as.integer(nrow(val_input) / BATCH_SIZE),
verbose = 1L,
callbacks = list(early_stopping)
)
I still have two questions regarding this: 1/ Why do I have to say shuffle = FALSE for the val_generator? When I tried with some test parameters and shuffle = TRUE it broke and retrieved this error:
> history <- model %>%
+ fit_generator(
+ generator = train_generator,
+ epochs = 100L,
+ steps_per_epoch = 2L, #as.integer(nrow(train_input) / BATCH_SIZE),
+ validation_data = val_generator,
+ validation_steps = 1L, #as.integer(nrow(val_input) / BATCH_SIZE),
+ verbose = 1L,
+ callbacks = list(early_stopping)
+ )
Epoch 1/100
1/2 [==============>...............] - ETA: 28s - loss: 0.7782 - dice_coef_loss_for_bce: 0.6753Error occurred in generator: cannot take a sample larger than the population when 'replace = FALSE'
Show Traceback
Rerun with Debug
Error in py_call_impl(callable, dots$args, dots$keywords) :
StopIteration:
2/ In the example unet_linux a py_iterator was used on top of their custom generator.
train_iterator <- py_iterator(train_generator(images_dir = images_dir,
masks_dir = masks_dir,
samples_index = train_index,
batch_size = batch_size))
Why is that if the parameters of fit_generator controls how many epochs should be run? My assumption was that the generator performs the operations and that py_iterator was enabling iterating again the generator. That's why I started with a combination of both, obviously i was wrong.
As a point 3/ I wanted to thank you very much for the link to the R blog it is very interesting! I am very much looking forward to your future articles!
Cheers
That's nice to hear, thank you!
Reg. 1), given the text of the error message it's very hard to imagine it could be anything different from batch_size
being bigger than nrow(data)
- for whatever reasons... Or indices
not being used, later, by some copy-paste thing? I'd try putting print()
statements everywhere, see what indices
is etc.
Reg. 2), I think that might be because the example was created before that py_iterator
stuff was taken care of internally.
Hi Sigrid, thank you for the detail.
I was trying over the past two days to run a pretty heavy training step (virtual size of the augmented data set 128000 images). The problem is that my RStudio is getting unstable and crashes at random times, making it impossible to complete the training step. My hyperparameters for training the model are the following:
BATCH_SIZE = 16L
EPOCHS = 100L
STEPS_PER_EPOCHS = 80L
VALIDATION_STEPS = as.integer(STEPS_PER_EPOCHS / 5L)
Training the model encompasses this:
train_generator <- custom_generator(data = train_input,
shuffle = TRUE,
batch_size = BATCH_SIZE)
val_generator <- custom_generator(data = val_input,
shuffle = FALSE,
batch_size = BATCH_SIZE)
early_stopping <- callback_early_stopping(patience = 8L)
history <- model %>%
fit_generator(
generator = train_generator,
epochs = EPOCHS,
steps_per_epoch = STEPS_PER_EPOCHS, #as.integer(nrow(train_input) / BATCH_SIZE),
validation_data = val_generator,
validation_steps = VALIDATION_STEPS, #as.integer(nrow(val_input) / (BATCH_SIZE / 2)),
verbose = 1L,
callbacks = list(early_stopping)
)
Any idea where this could be coming from?
Cheers
Try batch_size
= 1?
Weirdly enough if I use those parameters (for a quick testing):
BATCH_SIZE = 16L
EPOCHS = 30L
STEPS_PER_EPOCHS = 2L
VALIDATION_STEPS = 1L
it does run through without issues.
Also for all the training steps, I had ran before without data augmentation, were done with:
BATCH_SIZE = 32L
EPOCHS = 60L # up to 120
VALIDATION_PLIT = 0.2
and back then I had no problems.
If I would decrease batch_size to 1 wouldn't I be "under feeding" my model at this point?
Cheers
I think you need to experiment and find the best trade-off between speed and memory usage on your system (for this task)
Thanks a lot Sigrid, you confirmed my thoughts. I will try to just do that.
Dear all,
Following up on my semantic segmentation project already discussed here (Two classes semantic segmentation with U-Net) and here (weighting the different objects I am trying to identify).
For adding a bit of context I am willing to segment cells or nuclei from microscopy images and am providing for this as labels two masks (corresponding to my two classes): class one = the objects (nuclei or cells themselves) class two = the border of the objects only
As I have only limited amount of data with the ground truth segmentation (48 images and their corresponding masks). I would like to use data augmentation to improve the segmentation accuracy without over-fitting (basically like in the original publication from Reineberger et al.)
My model looks like this:
The compiled model:
With the corresponding metric:
My input data looks as following:
I am willing to use a 80% 20% split for my training and validation sets respectively.
Below is my attempt to augment the data:
However when I run this I get this error message (with traceback):
I do not know how to properly structured the data so that it would work. I have seen other people doing this in python but did not manage to replicate this in R.
My biggest concern and also the main problem is the following: I would like to augment X and apply the same spatial transformation to Y (like rotations, flips or crops) but no intensity / texture based transformation to Y (though I would like to apply this to X). This is also why in my example above I used only geometry based transformations.
Does someone know how to solve this problem?
Many thanks in advance.
Cheers,
Mathieu