how to prepare the input data in the folder HS_LiDAR_CNN?

[dzhong-hub]

Hi Danfeng, can you introduce how to prepare the input data for the HS_LiDAR_CNN? You mentioned "It is worth noting, however, that the patch centered by a pixel is usually used as the input of CNNs in RS image classification. For this reason, we need to extend the original image by the “replicate” operation, that is, copying the pixels within the image to that out of the original image boundary, to solve the problem of the boundaries of the multimodal RS data in the CNNs-related experiments." in your paper. my question is how to "replicate" from the original image? Your stepwise guidance would be very much appreciated. Thank you!

[danfenghong]

Hi Detang,

Thank you very much for your interest in our work! Nothing special, we just use the matlab function, i.e., padarray，to replicate the original image. PS: to our knowledge, you can choose the type of replicate that you would like, there is no big effect in the final performance.

Cheers, Danfeng

Detang Zhong @.***> 于2021年8月22日周日 下午7:33写道：

Hi Danfeng, can you introduce how to prepare the input data for the HS_LiDAR_CNN? You mentioned "It is worth noting, however, that the patch centered by a pixel is usually used as the input of CNNs in RS image classification. For this reason, we need to extend the original image by the “replicate” operation, that is, copying the pixels within the image to that out of the original image boundary, to solve the problem of the boundaries of the multimodal RS data in the CNNs-related experiments." in your paper. my question is how to "replicate" from the original image? Your stepwise guidance would be very much appreciated. Thank you!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/danfenghong/IEEE_TGRS_MDL-RS/issues/3, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFL2GZR2CM7MHAFKMUZU6PDT6DN7RANCNFSM5CS3QLMA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

[dzhong-hub]

Hi Danfeng,

thanks for your quick response! we are really interested in your many works. yes, padarray has options for the type of replicate. I noted that the HSI input data dimension for CNNs is (image size row x col) x (7x7xd (d=144 here)) = 12197x7056. How did you replicate the original image dimension 12197x144 to the input data dimension 12197x7056? I want to map our test data similarly to apply your code. Your help is very much appreciated!

[danfenghong]

No, no, it is not replicating the dimension. It is clear to see that 7056 is equal to 77144. I did not replicate the dimension. I just aim to store this input data conveniently, the input is 12197(the number of samples)77144, and I convey it to the 2D matrix, i.e., (121977056).

Detang Zhong @.***> 于2021年8月22日周日 下午11:05写道：

Hi Danfeng,

thanks for your quick response! we are really interested in your many works. yes, padarray has options for the type of replicate. I noted that the HSI input data dimension for CNNs is (image size row x col) x (7x7xd (d=144 here)) = 12197x7056. How did you replicate the original image dimension 12197x144 to the input data dimension 12197x7056? I want to map our test data similarly to apply your code. Your help is very much appreciated!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/danfenghong/IEEE_TGRS_MDL-RS/issues/3#issuecomment-903282560, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFL2GZU5J4VNFB53ZCWYFQ3T6EG2DANCNFSM5CS3QLMA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

[dzhong-hub]

it may be easier with an example. I have a rcb (example 627396162) original 3D HS image saved in a matrix A. How should I reformat and replicate it to a 2D r1*c1 matrix B that can be saved to a mat file "B.mat" and used as input data to your code.

Or I have done followings: % Convert 3d image data array to 2d first img2d = hyperConvert2d(img3d); % Normalize the 2d image data for i = 1 : size(img2d, 1)
img2d(i, :) = mat2gray(img2d(i, :)); end

Then what should I do to replicate the img2d so that your code can accept the replicated 2D array replicated_img2d? Many thanks!

[danfenghong]

You can also refer to the function in my another paper to see how to prepare the data. https://github.com/danfenghong/IEEE_TGRS_GCN/blob/master/DataGeneration_Functions/TR_TE_Generation2d_CNN.m

Detang Zhong @.***> 于2021年8月23日周一 上午10:26写道：

it may be easier with an example. I have a rcb (example 627396162) original 3D HS image saved in a matrix A. How should I reformat and replicate it to a 2D r1*c1 matrix B that can be saved to a mat file "B.mat" and used as input data to your code.

Or I have done followings: % Convert 3d image data array to 2d first img2d = hyperConvert2d(img3d); % Normalize the 2d image data for i = 1 : size(img2d, 1) img2d(i, :) = mat2gray(img2d(i, :)); end

Then what should I do to replicate the img2d so that your code can accept the replicated 2D array replicated_img2d? Many thanks!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/danfenghong/IEEE_TGRS_MDL-RS/issues/3#issuecomment-903395245, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFL2GZVJQCBNWJRCDMNCOETT6GWTRANCNFSM5CS3QLMA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

[dzhong-hub]

Hi Danfeng, thank you very much! The data generation code really help a lot.

[danfenghong]

Good to know! Hope it can help you! Cheers, Danfeng

Detang Zhong @.***> 于2021年8月24日周二 上午3:25写道：

Hi Danfeng, thank you very much! The data generation code really help a lot.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/danfenghong/IEEE_TGRS_MDL-RS/issues/3#issuecomment-904057525, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFL2GZT3KAA4UMYM5CNWSU3T6KOBXANCNFSM5CS3QLMA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

[dzhong-hub]

Hi Danfeng, many thanks for your support! I tried to test our own data. but I got a lot of dimension mismatch problems with the network training. I am not familiar with how to define the network. I noted that you hard coded the numbers of HSI and LiDAR, i.e. 144 and 21 respectively. I changed them to match our band numbers 217 for HSI and 5 for LiDAR. I also changed our number of training labels to 9 (your test case was 15). Can you give some guidance or point me to some references how to define the network to avoid the dimension mismatch problems? Or briefly explain the relationships between the band numbers and the layers of the network. Is there minimum number to the band number, for example, >16. In our case, LiDAR data has only 5 bands. Is this a problem? Your support are very much appreciated. Thank you!

[danfenghong]

You have to change the network input size to match your input, I guess. Cheers, Danfeng

Detang Zhong @.***> 于2021年9月10日周五 下午8:44写道：

Hi Danfeng, many thanks for your support! I tried to test our own data. but I got a lot of dimension mismatch problems with the network training. I am not familiar with how to define the network. I noted that you hard coded the numbers of HSI and LiDAR, i.e. 144 and 21 respectively. I changed them to match our band numbers 217 for HSI and 5 for LiDAR. I also changed our number of training labels to 9 (your test case was 15). Can you give some guidance or point me to some references how to define the network to avoid the dimension mismatch problems? Or briefly explain the relationships between the band numbers and the layers of the network. Is there minimum number to the band number, for example, >16. In our case, LiDAR data has only 5 bands. Is this a problem? Your support are very much appreciated. Thank you!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/danfenghong/IEEE_TGRS_MDL-RS/issues/3#issuecomment-916876465, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFL2GZQQMJW6X64H4B5ETITUBH4T3ANCNFSM5CS3QLMA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[dzhong-hub]

Thanks, I figured out that the issue is in my input data size. Now, the tests are on going. If I want to extend the network for 3 or 4 different modal datasets instead of two modalities, what should I pay attention to? Your supports are very much appreciated.

[danfenghong]

You have to add the network stream to 3 or 4 (now the stream is 2).

Detang Zhong @.***> 于2021年9月11日周六 下午9:55写道：

Thanks, I figured out that the issue is in my input data size. Now, the tests are on going. If I want to extend the network for 3 or 4 different modal datasets instead of two modalities, what should I pay attention to? Your supports are very much appreciated.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/danfenghong/IEEE_TGRS_MDL-RS/issues/3#issuecomment-917411730, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFL2GZQTMG5PLPG4QYNIW4TUBNNVLANCNFSM5CS3QLMA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[dzhong-hub]

Thank you, Danfeng! I will try it, but I am still learning more about your models. By the way, can you share your code or advise how to map the saved "feature" to the classified maps? Actually, I want verify our test results to get the classification accuracies. Your support are always appreciated.

[danfenghong]

For generating the classification maps, I used the Matlab to generate it. Maybe you can refer to https://github.com/danfenghong/IEEE_TGRS_GCN to generate the map.

Detang Zhong @.***> 于2021年9月19日周日 上午8:10写道：

Thank you, Danfeng! I will try it, but I am still learning more about your models. By the way, can you share your code or advise how to map the saved "feature" to the classified maps? Actually, I want verify our test results to get the classification accuracies. Your support are always appreciated.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/danfenghong/IEEE_TGRS_MDL-RS/issues/3#issuecomment-922392081, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFL2GZU3TKZNORBASIPZJ23UCUS7HANCNFSM5CS3QLMA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[dzhong-hub]

I tried to use the function draw_ClassificationMap.m there, but it didn't work to draw the predicted map. I put a question there too. Can you advise a little more clearly how to calculate the Pred_TE from the features? Thank you!

[danfenghong]

I think you have to see the codes corresponding to your features. Since now you have the trained model, it is very easy to directly input the testing samples into the model and you can obtain the predicted features.

Detang Zhong @.***> 于2021年9月19日周日 上午10:09写道：

I tried to use the function draw_ClassificationMap.m there, but it didn't work to draw the predicted map. I put a question there too. Can you advise a little more clearly how to calculate the Pred_TE from the features? Thank you!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/danfenghong/IEEE_TGRS_MDL-RS/issues/3#issuecomment-922403710, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFL2GZX35HOUQUI5FAS64U3UCVA7FANCNFSM5CS3QLMA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[dzhong-hub]

Hi Danfeng, I am just starting to learn TensorFlow, now I can get the trained model output, but no clue how to use it for prediction and evaluate it. It should be very straightforward. It looks to me, the following code line is doing the prediction, but it requires the test label (y_test_set) as input parameter. For a general prediction, it is not available. Are you familiar with the session.run() just for prediction without evaluation? Or can you help to define a prediction function based on the current network without the evaluation part, i.e. I just have the the input data x1 and x2, I want to get the predicted features from them?

feature, epoch_cost_dev, epoch_acc_dev = sess.run([joint_layerT, cost, accuracy], feed_dict={x1: x1_test_set, x2: x2_test_set, x1_full: x1_test_set, x2_full: x2_test_set, y: y_test_set, isTraining: False})

Thank you very much for your support in advance!

[danfenghong]

Hi,

I am now in China for holiday, and I am leaving from DLR. My private PC can not run the program now. The prediction function is quite simple. You can easily find them online.

Detang Zhong @.***> 于2021年9月21日周二 下午9:11写道：

Hi Danfeng, I am just starting to learn TensorFlow, now I can get the trained model output, but no clue how to use it for prediction and evaluate it. It should be very straightforward. It looks to me, the following code line is doing the prediction, but it requires the test label (y_test_set) as input parameter. For a general prediction, it is not available. Are you familiar with the session.run() just for prediction without evaluation? Or can you help to define a prediction function based on the current network without the evaluation part, i.e. I just have the the input data x1 and x2, I want to get the predicted features from them?

feature, epoch_cost_dev, epoch_acc_dev = sess.run([joint_layerT, cost, accuracy], feed_dict={x1: x1_test_set, x2: x2_test_set, x1_full: x1_test_set, x2_full: x2_test_set, y: y_test_set, isTraining: False})

Thank you very much for your support in advance!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/danfenghong/IEEE_TGRS_MDL-RS/issues/3#issuecomment-923973369, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFL2GZXGSB43O7OFCZ42HRLUDB76PANCNFSM5CS3QLMA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[dzhong-hub]

Thanks, Danfeng! We will figure it out.