search

tsdataclinic / PMLDataPipeline

A pipeline for detecting macro-plastics in coastal waters using Sentinel 2 data
Apache License 2.0
19 stars 13 forks source link

Error in running the download_features.py file. Can you please provide features.csv file? #1

Open gprashmi opened 3 years ago

gprashmi commented 3 years ago

Hi, I am working on the plastic detection project using Sentinel-2 satellite image and I am trying to run follow your repository to get the satellite image data. I created the features.csv file with the columns and label as mentioned. However, when I run the below command with my csv file, the code does not generate any data in the data folder, instead it is going to the exception command in download_features.py. I am getting the output as "Could not download feature" and "Could not process feature".

python download_features.py --features data/features.csv

Can you please provide me a sample features.csv file and it would be of great help for my project as I have limited time to complete it.

stuartlynn commented 3 years ago

Hey @gprashmi Thanks for your interest in the project. Would you be able to post or share the features.csv file that you are trying to use? That might help us debug the issue.

Just a quick question, have you checked that there are sentinel satellite passes at the times and locations of the samples you are producing the in features file?

gprashmi commented 3 years ago

Hi @stuartlynn, Please find the .csv file I am using for currently, this is for the location Mytilene, Greece. Also the Sentinel passes through this location at the given date.

features.csv

Please let me if this is the right way to form the .csv file.

stuartlynn commented 3 years ago

Great will take a look! Just another quick gut check, did you set up sentinel hub as described here:

https://github.com/tsdataclinic/PMLDataPipeline#sentinelhub

gprashmi commented 3 years ago

Yes, I did set-up the sentinel hub as described in there.

stuartlynn commented 3 years ago

Thanks! Just checking the obvious stuff. I should have time later today or tomorrow to troubleshoot this. Will let you know what I fin

gprashmi commented 3 years ago

Sure, thank you! Looking forward for your findings.

gprashmi commented 3 years ago

Hi @stuartlynn,

Another quick question, so I am using two indices: NDVI and FDI for detecting plastics and non-plastics. For my initial trial and understanding, I got the Sentinel-2 images in Google Earth Engine and applied the indices on the images. However I am unable to get a proper FDI/NDVI index values for plastics so that to label based on index values.

I have looked at the research paper "Finding Plastic Patches in Coastal Waters using Optical Satellite Data" to get the index values, but I am unable to distinguish plastics/non-plastics accurately with these values. Can you please let me know if there are resources that define FDI/NDVI/NDWI values for plastic pixels in the given image?

gprashmi commented 3 years ago

Hi @stuartlynn,

Did you get a chance to take a look at my .csv file?

Thank you in advance.

stuartlynn commented 3 years ago

Hey @gprashmi so I think what is happening is that your sentinel hub account isn't configured in quite the same way as ours. I think you need to manually set up two layers like follows:

image

I think if you are able to provide your sentinel-hub username I can also transfer you a copy of our config directly.

As for your general question about the indexes, I think the issue is that the indexes in the paper "Finding Plastic Patches in Coastal Waters using Optical Satellite Data" are calibrated for when a particular type of atmospheric correction is applied to the images which are probably different from the one being used in Google Earth Engine.

In addition, in the work we did, we found that the indexes alone were good at distinguishing between debris in general but you needed the trained model to properly distinguish plastics from other things like timber and spume.

Would love to hear more about the work you are trying to do. Happy to jump on a quick call as well if that helps to try and move it along.

gprashmi commented 3 years ago

Hi @stuartlynn,

Oh okay... so my features.csv file seems to be good? and once I set up these layers in my SentinlelHub, I should be able to run the features.csv file right?

gprashmi commented 3 years ago

Thank you for the information regarding the indexes. I have been trying to use the index value ranges from the paper and I am unable to distinguish plastics and non-plastics things and I am finding it difficult to find the index values (NDVI, NDWI, FDI) in other research papers/google. Is it possible to share the index values for plastics from your work?

Also it would be great help if we can have a quick call to help me move along.

stuartlynn commented 3 years ago

So in our work, we didn't use index ranges. We used the indexes themselves along with some of the raw band data to train our model using training data from PML.

The model it'self is included in the repo

https://github.com/tsdataclinic/PMLDataPipeline/tree/master/models/median_200_both_L1C

It might be worth trying to apply that model to your scenes to see if you are able to recover the plastic targets you have.

Ping me at stuart.lynn@twosigma.com and we can set up some time to chat!

stuartlynn commented 3 years ago

Oh okay... so my features.csv file seems to be good? and once I set up these layers in my SentinlelHub, I should be able to run the features.csv file right?

I believe so but if it doesn't let me know and we can troubleshoot some more.

gprashmi commented 3 years ago

So in our work, we didn't use index ranges. We used the indexes themselves along with some of the raw band data to train our model using training data from PML.

The model it'self is included in the repo

https://github.com/tsdataclinic/PMLDataPipeline/tree/master/models/median_200_both_L1C

It might be worth trying to apply that model to your scenes to see if you are able to recover the plastic targets you have.

Ping me at stuart.lynn@twosigma.com and we can set up some time to chat!

gprashmi commented 3 years ago

So in our work, we didn't use index ranges. We used the indexes themselves along with some of the raw band data to train our model using training data from PML.

The model it'self is included in the repo

https://github.com/tsdataclinic/PMLDataPipeline/tree/master/models/median_200_both_L1C

It might be worth trying to apply that model to your scenes to see if you are able to recover the plastic targets you have.

Ping me at stuart.lynn@twosigma.com and we can set up some time to chat!

Oh okay...Got it. Sure I will reach you at your email.

gprashmi commented 3 years ago

Oh okay... so my features.csv file seems to be good? and once I set up these layers in my SentinlelHub, I should be able to run the features.csv file right?

I believe so but if it doesn't let me know and we can troubleshoot some more.

Great, I have set up the layers in my SentinelHub account and will try running the command today and let you know how to goes.

gprashmi commented 3 years ago

Hi @stuartlynn,

As you mentioned, I set up the layers in the SentinelHub and tried to run the command: python download_features.py --features data/features.csv. However, I am facing the issue that the model seems to not enter the "LinearWorkflow" package seems to not run and I am getting the below error. I gave print "Entered" statements after entering the "process_feature" function and printed "Saving Done" after the SaveTask line. So its going into the exception as the next line: LinearWorkflow does not run. I also tried by giving the layer names I created in the cloud_classifier.py and input_tasks.py.

Screen Shot 2021-04-19 at 10 01 46 PM
gprashmi commented 3 years ago

Also, I have reached out to you on your email for a quick call. Please let me know what time works for you.