Error in d[idx_val, , , ] : The subscript out

[sue-shine]

When I run the following code for file 2_1 fit_CNNIDE_stage1,:

 `while(length(epoch_order) >= N_Batch) {

train_cov_pars <- (epoch >= (28L))       # Estimate cov. pars if epoch >= 28
count <- count + 1                       # increment batch number in epoch
idx <- epoch_order[1:N_Batch]            # Take first N_Batch
epoch_order <- epoch_order[-(1:N_Batch)] # Remove first batch from list
fd <- dict(data_in = d[idx,,,],          # Creat dictionary
           data_current = dfinal[idx,,,drop=F],
           data_future = dpred[idx,,,drop=F])

if(train_cov_pars) {
  TFrun <- run(list(trainnetcov, Cost1), feed_dict = fd)
} else {
  TFrun <- run(list(trainnet, Cost1), feed_dict = fd) # Train network on its own  
}

## Get Cost for this batch
Objective[[epoch]][count] <- TFrun[[2]]

## Every 10 samples do a random validation check
if((count %% 10) == 0) {
  idx_val <- sample(idxVal, N_Batch, replace = FALSE)
  fd <- dict(data_in = d[idx_val,,,],
             data_current = dfinal[idx_val,,,drop=F],
             data_future = dpred[idx_val,,,drop=F])
  Objective_val[[epoch]][count/10] <- run(Cost1, feed_dict = fd)
  cat(paste0("Epoch ", epoch, " ...  Step ", count, 
             " ... Cost: ", Objective_val[[epoch]][count/10], "\n"))
}

Epoch_train[epoch, ] <- c(mean(Objective[[epoch]]),
                          median(Objective[[epoch]]),
                          sd(Objective[[epoch]]),
                          mad(Objective[[epoch]]))

} } `

Error in d[idx_val, , , ] : The subscript out

How can I fix it? Thank you very much!

[andrewzm]

Strange, what is dim(d)? and what is idx_val when it crashes?

On Thu, Apr 8, 2021 at 3:47 PM sue-shine @.***> wrote:

When I run the following code for file 2_1 fit_CNNIDE_stage1,:

`while(length(epoch_order) >= N_Batch) {

train_cov_pars <- (epoch >= (28L)) # Estimate cov. pars if epoch >= 28 count <- count + 1 # increment batch number in epoch idx <- epoch_order[1:N_Batch] # Take first N_Batch epoch_order <- epoch_order[-(1:N_Batch)] # Remove first batch from list fd <- dict(data_in = d[idx,,,], # Creat dictionary data_current = dfinal[idx,,,drop=F], data_future = dpred[idx,,,drop=F])

if(train_cov_pars) { TFrun <- run(list(trainnetcov, Cost1), feed_dict = fd) } else { TFrun <- run(list(trainnet, Cost1), feed_dict = fd) # Train network on its own }

Get Cost for this batch

Objective[[epoch]][count] <- TFrun[[2]]

Every 10 samples do a random validation check

if((count %% 10) == 0) { idx_val <- sample(idxVal, N_Batch, replace = FALSE) fd <- dict(data_in = d[idx_val,,,], data_current = dfinal[idx_val,,,drop=F], data_future = dpred[idx_val,,,drop=F]) Objective_val[[epoch]][count/10] <- run(Cost1, feed_dict = fd) cat(paste0("Epoch ", epoch, " ... Step ", count, " ... Cost: ", Objective_val[[epoch]][count/10], "\n")) }

Epoch_train[epoch, ] <- c(mean(Objective[[epoch]]), median(Objective[[epoch]]), sd(Objective[[epoch]]), mad(Objective[[epoch]]))

} } `

Error in d[idx_val, , , ] : The subscript out

How can I fix it? Thank you very much!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/andrewzm/deepIDE/issues/2, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABEU3IXIT2ZX4R5QN7P3QTTHU7QJANCNFSM42SETPKQ .

[sue-shine]

The idx_val is training sets. The dim(d) is 601 349 60. I used the SST data from 2019-01-01 to 2019-03-01. And I changed the total number of time points and training + validation of time points in file CNN_fit_vars.R.

[sue-shine]

Can I set "nepochs" to a smaller value? What's the minimum I can set? Your code "nepochs<-30"

[andrewzm]

First try to run the code without changing anything. You will need TFv1.x. Your dim(d) doesn't look right to me... You should have 64x64 in your dimensions.

Andrew

On Thu, 8 Apr 2021, 20:17 sue-shine, @.***> wrote:

Can I set "nepochs" to a smaller value? What's the minimum I can set? Your code "nepochs<-30"

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/andrewzm/deepIDE/issues/2#issuecomment-815640876, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABEU3IDZBQWFYQ63TMVRETTHV7EHANCNFSM42SETPKQ .

[sue-shine]

Oh no sorry, my dim(d) is 1083 64 64 3. But my time points changed. What's the "TFv1.x" mean? Can I set "nepochs" to a smaller value? What's the minimum I can set? Your code "nepochs<-30"

[sue-shine]

My time points is 60, so I think the CNN_fit_vars.R in file common should be changed?

So I change "nT", "nT_Train_Val", "nT_Test". Nothing else has changed

[andrewzm]

Hi, I think you need to take some time to understand the code properly, for example nepochs is the "number of epochs" and you need to know what that is to know whether you can set it lower or not for your application. Also, TF = TensorFlow, the code will not work with TFv2 or above..

Andrew

On Thu, Apr 8, 2021 at 9:33 PM sue-shine @.***> wrote:

Oh no sorry, my dim(d) is 1083 64 64 3. But my time points changed. What's the "TFv1.x" mean? Can I set "nepochs" to a smaller value? What's the minimum I can set? Your code "nepochs<-30"

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/andrewzm/deepIDE/issues/2#issuecomment-815689445, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABEU3LLSXMCPPH22TXIQ7LTHWH7VANCNFSM42SETPKQ .

[sue-shine]

Excuse me, There are 25 time points in my data, and ran the code in the 2_1_fit_CNNIDE_stage1 files. But when I going on. Trained_Vars <- run(tf$trainable_variables(scope = NULL)) run <- tf$Session()$run All_Vars_tf <- tf$trainable_variables(scope = NULL) for(i in 1:length(All_Vars_tf)) { run(tf$assign(All_Vars_tf[[i]], Trained_Vars[[i]])) }

Error in d[idx_val, , , ] : The subscript out.

I checked that "All_Vars_tf" is a list of 36, and "Trained_Vars" is a list of 24.

[andrewzm]

I am pretty sure the code is correct.

It will take me sometime to find some bandwidth to re run it from top to bottom, I'll let you know when I do.

On Sat, 10 Apr 2021, 20:41 sue-shine, @.***> wrote:

Excuse me, There are 25 time points in my data, and ran the code in the 2_1_fit_CNNIDE_stage1 files. But when I going on. Trained_Vars <- run(tf$trainable_variables(scope = NULL)) run <- tf$Session()$run All_Vars_tf <- tf$trainable_variables(scope = NULL) for(i in 1:length(All_Vars_tf)) { run(tf$assign(All_Vars_tf[[i]], Trained_Vars[[i]])) }

Error in d[idx_val, , , ] : The subscript out.

I checked that "All_Vars_tf" is a list of 36, and "Trained_Vars" is a list of 24.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/andrewzm/deepIDE/issues/2#issuecomment-817115559, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABEU3K6GRCF5CBINYOKKSLTIATL3ANCNFSM42SETPKQ .

[sue-shine]

Thank you for your help!

[sue-shine]

Hi, I met a problem about the output of CNN, as the following figure: I can't understand what exactly CNN output refers to, Does it represent the flow of SST of the input sequense, Does this flow mean the same thing as the arrow with the varying kernel in the IDE model? Is the red contour resprensts the kernel? Is there any connection to the IDE model "the flow opposite to the shift of kernel "? Looking forward to your reply, Thanks!

[andrewzm]

Regarding your first question: w_i is the vector of all weights (i.e., contains w_ij for j = 1...r in it). The CNN outputs the entire vector.

Regarding your second question, the arrows are essentially the output of the IDE (the advection parameters). The contours are the process, and have nothing to do with the IDE kernel. I just chose a process which is made up of a kernel moving in space in time to make the outputted parameters easy to interpret. Regarding the phrase "the flow opposite to the shift of kernel", that has already been taken into account when drawing the arrows. How this helps.

Best, Andrew

On Fri, 23 Apr 2021, 21:17 sue-shine, @.***> wrote:

Hi, I met a problem about the output of CNN, as the following figure: [image: 1] https://user-images.githubusercontent.com/56682166/115862020-2936aa80-a466-11eb-9230-dc98c9940b2d.jpg I can't understand what exactly CNN output refers to, Does it represent the flow of SST of the input sequense, Does this flow mean the same thing as the arrow with the varying kernel in the IDE model? Is the red contour resprensts the kernel? Is there any connection to the IDE model "the flow opposite to the shift of kernel "? Looking forward to your reply, Thanks!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/andrewzm/deepIDE/issues/2#issuecomment-825588023, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABEU3NTPSUJ4EZCWQHC6EDTKFJNNANCNFSM42SETPKQ .

[sue-shine]

Oh I see, Thanks for you patience! Do the observed data in CNNIDE model have to be normally distributed? Does this model work if my response variables follow a binomial distribution, such as the rate of disease infection?

[andrewzm]

Hi, We have finished rebuilding a system from scratch using TensorFlow 1.12 as mentioned in the README, and the scripts are running fine (I stopped it at nepoch == 17, no use running further). So I think there could be something wrong with your system installation, and I cannot help you there sorry.

The CNN-IDE takes in data on a spatio-temporal grid and outputs parameters associated with the diffusion-advection equation. The EnKF allows you to have missing data, and noisy data, but Gaussianity is assumed. You would need to change the EnKF to take discrete data and also a link function so that the CNN is operating on the transformed space. It's not that straightforward though, and probably another paper in itself.

Andrew

On Sun, Apr 25, 2021 at 5:10 PM sue-shine @.***> wrote:

Oh I see, Thanks for you patience! Do the observed data in CNNIDE model have to be normally distributed? Does this model work if my response variables follow a binomial distribution, such as the rate of disease infection?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/andrewzm/deepIDE/issues/2#issuecomment-826271643, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABEU3MY56PQCJY2SSVWR7DTKO55NANCNFSM42SETPKQ .