search

samapriya / Planet-GEE-Pipeline-CLI

Planet and Google Earth Engine Pipeline Command Line Interface Tool
https://pypi.org/project/ppipe/
Apache License 2.0
42 stars 12 forks source link

Error with Metadata Parser #16

Closed katherinehess closed 4 years ago

katherinehess commented 4 years ago

Hi Samapriya,

I'll be embarrassed if this is another non-issue, but I've really exhausted my resources on this one...

First, it says a few places in the documentation that I can skip the metadata input for selupload if I am okay with not having it in GEE, so I tried but when I try to run the command:

ppipe selupload --source "Y:\duncansongp\Planet\fulldata" --dest "users/katehess1/planetforest" -u katehess2@gmail.com

It asks for and takes my password, but then I get this error:

File "c:\users\katehess\appdata\local\continuum\miniconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.SessionNotCreatedException: Message: Unable to find a matching set of capabilities

Could this be because I am not providing metadata? So then I tried using the metadata parser tool (command below), directing it to a folder that only contains XMLs (which were downloaded using porder), and got the error shown below.

ppipe metadata --asset PS4B --mf "Y:\duncansongp\Planet\xmls" --mfile "Y:\duncansongp\Planet\metadata\mdata.csv" --errorlog "Y:\duncansongp\Planet\metadata\errlog.csv"

File "c:\users\katehess\appdata\local\continuum\miniconda3\lib\csv.py", line 155, in writerow return self.writer.writerow(self._dict_to_list(rowdict)) TypeError: a bytes-like object is required, not 'str'

I'm going to keep troubleshooting, but I wanted to reach out and see if you have any idea what could be the problem here. Really appreciate any help!

I've attached an XML that I tried to put through the metadata tool.

20180501_082515_0f34_3B_AnalyticMS_metadata.zip

samapriya commented 4 years ago

Hey @katherinehess , The tool uses selenium to upload your tiff files to your Google Earth Engine accounts. If you are using ppipe on a server without a screen, this will not work and neither in a headless setup.

As far as your second issue with the metadata tool, I was able to reproduce it and I will release a new version with some much-needed updates and improvements. Can you give me some background about your setup, are you using your personal machine or a server setup? your operating system and so on this will help me trace the selenium issue further as needed.

katherinehess commented 4 years ago

Hi @samapriya ,

I had a breakthrough but am now having a different error- I saw selenium was related to firefox which I did not have installed, and installing it seems to have fixed the selenium error.

But now I run the following selupload command and get the result that the upload has failed, but with no indication of why: ppipe selupload --source "Y:\duncansongp\Planet\fulldata" --dest "users/katehess/planetforests" -u katehess@terpmail.umd.edu -mf PS4B -m "Y:\duncansongp\Planet\metadata\md.csv" Password: Collection users/katehess/planetforests already exists Processing image 1 out of 1: Y:\duncansongp\Planet\fulldata\20160108_035841_0c73_3B_AnalyticMS.tif Upload of 20160108_035841_0c73_3B_AnalyticMS has failed.

I am using my own machine to run the command, so everything is being processed locally. The tiffs I am trying to upload are stored on a server (the Y: drive), but I've tried running it with them on my local C: drive and the result has been the same.

I have Windows 10 Enterprise (version 10.0.17134.95) and I'm using an Anaconda prompt instead of the command prompt. Please let me know if you need any other details!

katherinehess commented 4 years ago

The images I am trying to upload are each around ~100,000KB so github won't let me attach an example, but I'm including an older clipped image and a .csv file with some random info in it (just to give the program something to work with for the metadata input). Interestingly when running this one I get a new error message: Collection users/katehess/planetforests already exists Processing image 1 out of 1: C:\localplanetdata\20170830_150012_0f4e_3B_AnalyticMS_clip.tif invalid syntax (, line 2) Invalid URL 'None': No schema supplied. Perhaps you meant http://None? Upload of 20170830_150012_0f4e_3B_AnalyticMS_clip has failed.

planettest.zip

samapriya commented 4 years ago

Issues fixed in v0.4.7 including both the metadata parser and the uploader. Use geeup for the generalized upload of both raster and tables.