search

RETURN-project / makeDataCube

Data management
Apache License 2.0
0 stars 0 forks source link

Buggy vignette in master #5

Closed PabRod closed 4 years ago

PabRod commented 4 years ago

I am not managing to fully execute vignettes/make_Landsat_cube.Rmd.

The second and last Python chunk throws the error below.

OperationalError: no such table: catalog

Detailed traceback: 
  File "<string>", line 7, in <module>
  File "/home/pablo/anaconda3/lib/python3.7/site-packages/pylandsat/catalog.py", line 99, in search
    params=(geom, begin, end, maxcloud, sensors, tiers, geom))
  File "/home/pablo/anaconda3/lib/python3.7/site-packages/pylandsat/database.py", line 72, in query
    c.execute(query, params)

I'm trying to figure out what's the problem. Do you have any clue, @wandadk?

wandadk commented 4 years ago

Yes, I think that this has to do with the code on line 148, where the catalog is synced. I need to check, but I suspect that the catalog is placed at a wrong location. I believe this needs to be located in the temporary folder.

system("pylandsat sync-database", intern = TRUE, ignore.stderr = TRUE)

wandadk commented 4 years ago

No, I was wrong. With the following command, you can check where the catalog should be located: pylandsat print-data-dir

For me, that is /home/wanda/.local/share/pylandsat, and the catalog is called landsat.db

wandadk commented 4 years ago

@PabRod Can you maybe check if you were able to download the catalog? The file should be around 1.4 G. Thanks!

PabRod commented 4 years ago

Thanks for your quick responses!

I see the problem. The file is there, but it is empty. Is it supposed to be downloaded externally or through the script?

PabRod commented 4 years ago

I just noticed I didn't create the .laads file. I'm doing it now.

wandadk commented 4 years ago

Good that you noticed this! The .laads file is for the WVP data, so I guess that the problem is related to something else. What happens if you run the following commands: system("pylandsat sync-database", intern = TRUE, ignore.stderr = TRUE) or: system("pylandsat sync-database -f", intern = TRUE, ignore.stderr = TRUE)

The last one should force the catalog to sync. The sync process can take some time (at least on my laptop).

wandadk commented 4 years ago

Thanks for your quick responses!

I see the problem. The file is there, but it is empty. Is it supposed to be downloaded externally or through the script?

The catalog is downloaded via the pylandsat package (python), using the following commands: pylandsat sync-database or pylandsat sync-database -f if you want to force the download. This package needs to be installed in order to download the data.

PabRod commented 4 years ago

Certainly the problem is related to this.

Before, this command took only a few seconds to run (and didn't throw any error after downloading an empty database). Now, it takes much longer (I am indeed waiting right now, it is supposed to be ready in ~20 min).

Thanks!

wandadk commented 4 years ago

Ok, great! I'm not entirely sure how to solve this bug. Running the command via the terminal always worked for me. Maybe it has something to do with the settings of the system() call?

PabRod commented 4 years ago

TL;DR: it was my fault.

I did something wrong with the .laads file, and pylandsat sync-database doesn't throw an error if the .laads file is missing. I will write a snippet that checks this to avoid this problem to future users.

wandadk commented 4 years ago

Ok, thanks! Great that we found the problem :-).