Landsat ingest - Githubissues

Maite-Machipi commented 6 years ago

hi @boney-bun ,

can you please assist regarding the Landsat ingest the metadata is a txt file and i believe the ingest looks for xml instead of txt, looking at the past ingested data it looks like it was converted to xml. please assist with the conversion.

boney-bun commented 6 years ago

Hi @Maite-Machipi,

Yes, the ingestor will search for xml metadata. The current system has no a txt to xml conversion yet.

Is there a reason why we would like to ingest txt metadata? Since a txt is not a common format for metadata.

Maite-Machipi commented 6 years ago

hi @boney-bun

our Landsat metadata comes in txt instead of xml. thats why im assuming that previously it might have been converted.

boney-bun commented 6 years ago

Good morning @Maite-Machipi

I've been looking at the past codes. It seems that the metadata comes with xml and txt files some time in the past. The current implementation requires only a xml metadata.

Do you have a set of xml and txt example to look at? I need to compare and understand the metadata if we are going to do the conversion.

boney-bun commented 6 years ago

Hi @Maite-Machipi

I have developed the landsat ingestor using txt metadata (#540). I put them on the production so that you can also test the code.

Some issues and confusions:

I'm not sure whether the product name is correct. Could you verify it?
some text files have MTL suffix, others are ANG. Both have different structure in content. the txt ingestor is focusing in MTL.
the file naming inconsistent: some are separated by many underscore, the others are only with one underscore.
some text files seems to have incorrect name. instead of L0(zero)8, the names start with LO(letter o)8. which one is correct?
hardcoded fields (needs to be populated from metadata):
- solar_zenith_angle
- solar_azimuth_angle
- band_count
- orbit_number
I notice for a 9 bands data, there are 9 different images. Which one is to be used as the thumbnail? for now, the thumbnail uses an image ends with _BQA.TIF.
I think the code still need to be tested on xml metadata as well. please let me know if you have xml metadata. I'm more than happy to help you on ingesting these metadata.

Instead of developing new code, I decided to make use the existing implementation. To ingest metadata, you can now specify -u. Run the following command to ingest txt metadata:

python manage.py landsat_harvest -u

If the thumbnails for all metadata are not fully available, we can simply ignore the missing thumbnails by adding the -i argument. Then, the command is: python manage.py landsat_harvest -u -i

Running the code in my machine:

root@uwsgi:/home/web/django_project# python manage.py landsat_harvest -u -i
{'WORKER_ENV_C_FORCE_ROOT': 'true', 'VIRTUAL_HOST': 'catalogue.kartoza.com', 'CATALOGUE_UWSGI_ENV_VIRTUAL_PORT': '8080', 'DB_ENV_USERNAME': 'docker', 'RABBITMQ_1_PORT_5672_TCP_PORT': '5672', 'RABBITMQ_PORT_4369_TCP_ADDR': '172.17.0.5', 'UWSGI_PORT_8080_TCP': 'tcp://172.17.0.7:8080', 'CATALOGUE_RABBITMQ_1_ENV_USER': 'rabbit_user', 'CATALOGUE_DB_PORT': 'tcp://172.17.0.2:5432', 'WORKER_ENV_PYTHON_PIP_VERSION': '10.0.1', 'CATALOGUE_UWSGI_ENV_LANG': 'C.UTF-8', 'CATALOGUE_WORKER_1_NAME': '/catalogue_uwsgi_run_13/catalogue_worker_1', 'WORKER_1_ENV_DJANGO_SETTINGS_MODULE': 'core.settings.prod_docker', 'UWSGI_ENV_LANG': 'C.UTF-8', 'PYTHON_VERSION': '2.7.14', 'CATALOGUE_SMTP_ENV_DEBIAN_FRONTEND': 'noninteractive', 'CATALOGUE_RABBITMQ_1_PORT_5672_TCP_ADDR': '172.17.0.5', 'SMTP_ENV_maildomain': 'kartoza.com', 'WORKER_1_PORT_8080_TCP_PORT': '8080', 'CATALOGUE_RABBITMQ_1_PORT_5671_TCP': 'tcp://172.17.0.5:5671', 'CATALOGUE_RABBITMQ_1_PORT_25672_TCP_ADDR': '172.17.0.5', 'CATALOGUE_RABBITMQ_1_ENV_RABBITMQ_GPG_KEY': '0A9AF2115F4687BD29803A206B73A36E6026DFCA', 'DB_PORT_5432_TCP_PROTO': 'tcp', 'WORKER_NAME': '/catalogue_uwsgi_run_13/worker', 'WORKER_ENV_LANG': 'C.UTF-8', 'RABBITMQ_PORT_5672_TCP_PROTO': 'tcp', 'RABBITMQ_1_ENV_USER': 'rabbit_user', 'UWSGI_ENV_DATABASE_USERNAME': 'docker', 'RABBITMQ_PORT': 'tcp://172.17.0.5:4369', 'CATALOGUE_RABBITMQ_1_ENV_RABBITMQ_SASL_LOGS': '-', 'RABBITMQ_HOST': 'rabbitmq', 'RABBITMQ_1_PORT_5671_TCP_PROTO': 'tcp', 'CATALOGUE_WORKER_1_ENV_DATABASE_PASSWORD': 'docker', 'CATALOGUE_UWSGI_ENV_DATABASE_NAME': 'gis', 'CATALOGUE_UWSGI_ENV_GPG_KEY': 'C01E1CAD5EA2C4F0B8E3571504C367C218ADD4FF', 'CATALOGUE_DB_ENV_no_proxy': '*.local, 169.254/16', 'CATALOGUE_RABBITMQ_1_ENV_GOSU_VERSION': '1.10', 'RABBITMQ_1_PORT_25672_TCP': 'tcp://172.17.0.5:25672', 'DB_NAME': '/catalogue_uwsgi_run_13/db', 'CATALOGUE_WORKER_1_ENV_DEBIAN_FRONTEND': 'noninteractive', 'RABBITMQ_1_ENV_GOSU_VERSION': '1.10', 'DB_ENV_DEBIAN_FRONTEND': 'noninteractive', 'CATALOGUE_UWSGI_NAME': '/catalogue_uwsgi_run_13/catalogue-uwsgi', 'RABBITMQ_1_ENV_RABBITMQ_GITHUB_TAG': 'v3.7.4', 'CATALOGUE_UWSGI_ENV_DATABASE_HOST': 'db', 'WORKER_1_ENV_VIRTUAL_HOST': 'catalogue.kartoza.org', 'RABBITMQ_ENV_GOSU_VERSION': '1.10', 'CATALOGUE_WORKER_1_PORT_8080_TCP_ADDR': '172.17.0.6', 'RABBITMQ_PORT_4369_TCP': 'tcp://172.17.0.5:4369', 'RABBITMQ_1_PORT_4369_TCP_PROTO': 'tcp', 'CATALOGUE_WORKER_1_ENV_PYTHON_PIP_VERSION': '10.0.1', 'CATALOGUE_WORKER_1_PORT_8080_TCP_PORT': '8080', 'RABBITMQ_1_ENV_RABBITMQ_SASL_LOGS': '-', 'HOSTNAME': 'uwsgi', 'CATALOGUE_WORKER_1_ENV_VIRTUAL_PORT': '8080', 'RABBITMQ_ENV_RABBIT_PASSWORD': 'rabbit_test_password', 'RABBITMQ_1_PORT_25672_TCP_PORT': '25672', 'CATALOGUE_UWSGI_PORT_8080_TCP_ADDR': '172.17.0.7', 'DATABASE_HOST': 'db', 'RABBITMQ_NAME': '/catalogue_uwsgi_run_13/rabbitmq', 'CATALOGUE_UWSGI_ENV_DATABASE_PASSWORD': 'docker', 'CATALOGUE_RABBITMQ_1_ENV_no_proxy': '*.local, 169.254/16', 'DB_PORT_5432_TCP_PORT': '5432', 'DB_PORT': 'tcp://172.17.0.2:5432', 'CATALOGUE_DB_PORT_5432_TCP_ADDR': '172.17.0.2', 'UWSGI_PORT_8080_TCP_PORT': '8080', 'WORKER_PORT_8080_TCP': 'tcp://172.17.0.6:8080', 'RABBITMQ_PORT_25672_TCP_ADDR': '172.17.0.5', 'WORKER_1_ENV_DATABASE_NAME': 'gis', 'CATALOGUE_WORKER_1_PORT': 'tcp://172.17.0.6:8080', 'RABBITMQ_1_PORT_4369_TCP_PORT': '4369', 'CATALOGUE_DB_NAME': '/catalogue_uwsgi_run_13/catalogue-db', 'RABBITMQ_1_PORT_5672_TCP_PROTO': 'tcp', 'RABBITMQ_1_PORT_4369_TCP_ADDR': '172.17.0.5', 'SHLVL': '1', 'RABBITMQ_PORT_25672_TCP': 'tcp://172.17.0.5:25672', 'CATALOGUE_RABBITMQ_1_PORT_25672_TCP': 'tcp://172.17.0.5:25672', 'WORKER_1_ENV_PYTHON_VERSION': '2.7.14', 'CATALOGUE_RABBITMQ_1_PORT_5671_TCP_ADDR': '172.17.0.5', 'UWSGI_ENV_GPG_KEY': 'C01E1CAD5EA2C4F0B8E3571504C367C218ADD4FF', 'CATALOGUE_RABBITMQ_1_PORT_5672_TCP_PROTO': 'tcp', 'SMTP_ENV_smtp_user': 'noreply:docker', 'UWSGI_ENV_RABBITMQ_HOST': 'rabbitmq', 'WORKER_ENV_GPG_KEY': 'C01E1CAD5EA2C4F0B8E3571504C367C218ADD4FF', 'RABBITMQ_PORT_5672_TCP_PORT': '5672', 'UWSGI_ENV_VIRTUAL_HOST': 'catalogue.kartoza.com', 'UWSGI_PORT_8080_TCP_PROTO': 'tcp', 'RABBITMQ_ENV_RABBITMQ_LOGS': '-', 'CATALOGUE_UWSGI_PORT': 'tcp://172.17.0.7:8080', 'WORKER_1_ENV_RABBITMQ_HOST': 'rabbitmq', 'WORKER_PORT_8080_TCP_PROTO': 'tcp', 'VIRTUAL_PORT': '8080', 'RABBITMQ_PORT_5671_TCP_ADDR': '172.17.0.5', 'RABBITMQ_PORT_4369_TCP_PORT': '4369', 'RABBITMQ_1_ENV_RABBITMQ_LOGS': '-', 'WORKER_1_ENV_DATABASE_USERNAME': 'docker', 'WORKER_ENV_DJANGO_SETTINGS_MODULE': 'core.settings.prod_docker', 'WORKER_1_ENV_no_proxy': '*.local, 169.254/16', '_': '/usr/local/bin/python', 'WORKER_ENV_DATABASE_NAME': 'gis', 'RABBITMQ_1_ENV_RABBITMQ_GPG_KEY': '0A9AF2115F4687BD29803A206B73A36E6026DFCA', 'RABBITMQ_ENV_LANG': 'C.UTF-8', 'CATALOGUE_WORKER_1_ENV_C_FORCE_ROOT': 'true', 'CATALOGUE_RABBITMQ_1_ENV_RABBITMQ_VERSION': '3.7.4', 'RABBITMQ_PORT_5672_TCP_ADDR': '172.17.0.5', 'DATABASE_NAME': 'gis', 'WORKER_1_NAME': '/catalogue_uwsgi_run_13/worker_1', 'RABBITMQ_ENV_RABBITMQ_GITHUB_TAG': 'v3.7.4', 'WORKER_ENV_RABBITMQ_HOST': 'rabbitmq', 'WORKER_1_ENV_C_FORCE_ROOT': 'true', 'CATALOGUE_RABBITMQ_1_PORT_25672_TCP_PORT': '25672', 'DJANGO_SETTINGS_MODULE': 'core.settings.prod_docker', 'CATALOGUE_UWSGI_ENV_DEBIAN_FRONTEND': 'noninteractive', 'WORKER_1_PORT_8080_TCP_ADDR': '172.17.0.6', 'WORKER_ENV_DEBIAN_FRONTEND': 'noninteractive', 'UWSGI_ENV_DATABASE_HOST': 'db', 'UWSGI_ENV_DATABASE_PASSWORD': 'docker', 'CATALOGUE_SMTP_ENV_smtp_user': 'noreply:docker', 'UWSGI_PORT': 'tcp://172.17.0.7:8080', 'CATALOGUE_UWSGI_ENV_PYTHON_VERSION': '2.7.14', 'RABBITMQ_PORT_5671_TCP_PORT': '5671', 'RABBITMQ_PORT_5671_TCP': 'tcp://172.17.0.5:5671', 'HOME': '/root', 'CATALOGUE_RABBITMQ_1_ENV_RABBITMQ_DEBIAN_VERSION': '3.7.4-1', 'CATALOGUE_RABBITMQ_1_PORT': 'tcp://172.17.0.5:4369', 'LANG': 'C.UTF-8', 'RABBITMQ_1_ENV_RABBIT_PASSWORD': 'rabbit_test_password', 'CATALOGUE_RABBITMQ_1_PORT_4369_TCP_ADDR': '172.17.0.5', 'UWSGI_ENV_DEBIAN_FRONTEND': 'noninteractive', 'RABBITMQ_ENV_RABBITMQ_VERSION': '3.7.4', 'CATALOGUE_RABBITMQ_1_PORT_4369_TCP_PORT': '4369', 'PYTHON_PIP_VERSION': '10.0.1', 'RABBITMQ_PORT_25672_TCP_PORT': '25672', 'CATALOGUE_WORKER_1_ENV_PYTHON_VERSION': '2.7.14', 'CATALOGUE_UWSGI_PORT_8080_TCP_PROTO': 'tcp', 'SMTP_ENV_DEBIAN_FRONTEND': 'noninteractive', 'RABBITMQ_PORT_4369_TCP_PROTO': 'tcp', 'RABBITMQ_1_PORT_25672_TCP_PROTO': 'tcp', 'CATALOGUE_DB_ENV_USERNAME': 'docker', 'CATALOGUE_WORKER_1_ENV_DJANGO_SETTINGS_MODULE': 'core.settings.prod_docker', 'RABBITMQ_1_ENV_RABBITMQ_NODENAME': 'rabbit', 'UWSGI_ENV_no_proxy': '*.local, 169.254/16', 'RABBITMQ_1_ENV_no_proxy': '*.local, 169.254/16', 'CATALOGUE_RABBITMQ_1_PORT_4369_TCP_PROTO': 'tcp', 'CATALOGUE_RABBITMQ_1_PORT_5671_TCP_PROTO': 'tcp', 'CATALOGUE_WORKER_1_ENV_LANG': 'C.UTF-8', 'RABBITMQ_1_PORT_5672_TCP': 'tcp://172.17.0.5:5672', 'RABBITMQ_1_PORT_5671_TCP': 'tcp://172.17.0.5:5671', 'DEBIAN_FRONTEND': 'noninteractive', 'WORKER_ENV_PYTHON_VERSION': '2.7.14', 'CATALOGUE_RABBITMQ_1_PORT_25672_TCP_PROTO': 'tcp', 'WORKER_1_PORT': 'tcp://172.17.0.6:8080', 'WORKER_ENV_no_proxy': '*.local, 169.254/16', 'CATALOGUE_UWSGI_ENV_PYTHON_PIP_VERSION': '10.0.1', 'CATALOGUE_UWSGI_ENV_RABBITMQ_HOST': 'rabbitmq', 'RABBITMQ_1_ENV_LANG': 'C.UTF-8', 'CATALOGUE_WORKER_1_ENV_DATABASE_USERNAME': 'docker', 'DB_PORT_5432_TCP': 'tcp://172.17.0.2:5432', 'CATALOGUE_WORKER_1_PORT_8080_TCP': 'tcp://172.17.0.6:8080', 'WORKER_PORT_8080_TCP_ADDR': '172.17.0.6', 'SMTP_NAME': '/catalogue_uwsgi_run_13/smtp', 'WORKER_PORT_8080_TCP_PORT': '8080', 'WORKER_1_ENV_LANG': 'C.UTF-8', 'CATALOGUE_RABBITMQ_1_NAME': '/catalogue_uwsgi_run_13/catalogue_rabbitmq_1', 'RABBITMQ_PORT_25672_TCP_PROTO': 'tcp', 'WORKER_1_PORT_8080_TCP': 'tcp://172.17.0.6:8080', 'CATALOGUE_WORKER_1_PORT_8080_TCP_PROTO': 'tcp', 'CATALOGUE_RABBITMQ_1_PORT_5671_TCP_PORT': '5671', 'UWSGI_ENV_PYTHON_PIP_VERSION': '10.0.1', 'RABBITMQ_1_PORT_5672_TCP_ADDR': '172.17.0.5', 'CATALOGUE_SMTP_ENV_maildomain': 'kartoza.com', 'RABBITMQ_1_PORT_25672_TCP_ADDR': '172.17.0.5', 'CATALOGUE_RABBITMQ_1_PORT_5672_TCP_PORT': '5672', 'DB_ENV_no_proxy': '*.local, 169.254/16', 'RABBITMQ_ENV_RABBITMQ_DEBIAN_VERSION': '3.7.4-1', 'UWSGI_ENV_VIRTUAL_PORT': '8080', 'CATALOGUE_RABBITMQ_1_ENV_RABBITMQ_GITHUB_TAG': 'v3.7.4', 'WORKER_ENV_VIRTUAL_PORT': '8080', 'DB_ENV_PASS': 'docker', 'UWSGI_NAME': '/catalogue_uwsgi_run_13/uwsgi', 'CATALOGUE_DB_ENV_DEBIAN_FRONTEND': 'noninteractive', 'WORKER_ENV_DATABASE_USERNAME': 'docker', 'CATALOGUE_DB_PORT_5432_TCP_PROTO': 'tcp', 'WORKER_PORT': 'tcp://172.17.0.6:8080', 'UWSGI_ENV_DJANGO_SETTINGS_MODULE': 'core.settings.prod_docker', 'CATALOGUE_UWSGI_PORT_8080_TCP_PORT': '8080', 'RABBITMQ_ENV_RABBITMQ_GPG_KEY': '0A9AF2115F4687BD29803A206B73A36E6026DFCA', 'UWSGI_PORT_8080_TCP_ADDR': '172.17.0.7', 'CATALOGUE_RABBITMQ_1_PORT_5672_TCP': 'tcp://172.17.0.5:5672', 'PATH': '/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'CATALOGUE_WORKER_1_ENV_DATABASE_NAME': 'gis', 'RABBITMQ_PORT_5672_TCP': 'tcp://172.17.0.5:5672', 'CATALOGUE_SMTP_NAME': '/catalogue_uwsgi_run_13/catalogue-smtp', 'CATALOGUE_WORKER_1_ENV_DATABASE_HOST': 'db', 'RABBITMQ_1_ENV_RABBITMQ_DEBIAN_VERSION': '3.7.4-1', 'TERM': 'xterm', 'RABBITMQ_ENV_RABBITMQ_SASL_LOGS': '-', 'RABBITMQ_1_PORT_5671_TCP_ADDR': '172.17.0.5', 'CATALOGUE_UWSGI_ENV_VIRTUAL_HOST': 'catalogue.kartoza.com', 'CATALOGUE_WORKER_1_ENV_GPG_KEY': 'C01E1CAD5EA2C4F0B8E3571504C367C218ADD4FF', 'CATALOGUE_UWSGI_ENV_no_proxy': '*.local, 169.254/16', 'CATALOGUE_RABBITMQ_1_ENV_RABBITMQ_NODENAME': 'rabbit', 'WORKER_1_PORT_8080_TCP_PROTO': 'tcp', 'RABBITMQ_1_PORT': 'tcp://172.17.0.5:4369', 'CATALOGUE_WORKER_1_ENV_no_proxy': '*.local, 169.254/16', 'CATALOGUE_RABBITMQ_1_ENV_RABBITMQ_LOGS': '-', 'WORKER_1_ENV_DEBIAN_FRONTEND': 'noninteractive', 'CATALOGUE_DB_PORT_5432_TCP_PORT': '5432', 'WORKER_ENV_DATABASE_PASSWORD': 'docker', 'RABBITMQ_ENV_USER': 'rabbit_user', 'CATALOGUE_RABBITMQ_1_ENV_RABBIT_PASSWORD': 'rabbit_test_password', 'CATALOGUE_SMTP_ENV_no_proxy': '*.local, 169.254/16', 'GPG_KEY': 'C01E1CAD5EA2C4F0B8E3571504C367C218ADD4FF', 'WORKER_1_ENV_VIRTUAL_PORT': '8080', 'SMTP_ENV_no_proxy': '*.local, 169.254/16', 'WORKER_ENV_DATABASE_HOST': 'db', 'UWSGI_ENV_DATABASE_NAME': 'gis', 'DATABASE_USERNAME': 'docker', 'CATALOGUE_WORKER_1_ENV_RABBITMQ_HOST': 'rabbitmq', 'WORKER_1_ENV_PYTHON_PIP_VERSION': '10.0.1', 'DATABASE_PASSWORD': 'docker', 'DB_PORT_5432_TCP_ADDR': '172.17.0.2', 'CATALOGUE_UWSGI_ENV_DJANGO_SETTINGS_MODULE': 'core.settings.prod_docker', 'RABBITMQ_PORT_5671_TCP_PROTO': 'tcp', 'WORKER_ENV_VIRTUAL_HOST': 'catalogue.kartoza.org', 'RABBITMQ_1_PORT_4369_TCP': 'tcp://172.17.0.5:4369', 'UWSGI_ENV_PYTHON_VERSION': '2.7.14', 'CATALOGUE_UWSGI_ENV_DATABASE_USERNAME': 'docker', 'CATALOGUE_RABBITMQ_1_ENV_LANG': 'C.UTF-8', 'CATALOGUE_RABBITMQ_1_PORT_4369_TCP': 'tcp://172.17.0.5:4369', 'RABBITMQ_1_PORT_5671_TCP_PORT': '5671', 'RABBITMQ_ENV_RABBITMQ_NODENAME': 'rabbit', 'WORKER_1_ENV_GPG_KEY': 'C01E1CAD5EA2C4F0B8E3571504C367C218ADD4FF', 'CATALOGUE_UWSGI_PORT_8080_TCP': 'tcp://172.17.0.7:8080', 'WORKER_1_ENV_DATABASE_HOST': 'db', 'WORKER_1_ENV_DATABASE_PASSWORD': 'docker', 'no_proxy': '*.local, 169.254/16', 'PWD': '/home/web/django_project', 'RABBITMQ_1_NAME': '/catalogue_uwsgi_run_13/rabbitmq_1', 'RABBITMQ_1_ENV_RABBITMQ_VERSION': '3.7.4', 'CATALOGUE_WORKER_1_ENV_VIRTUAL_HOST': 'catalogue.kartoza.org', 'RABBITMQ_ENV_no_proxy': '*.local, 169.254/16', 'CATALOGUE_DB_ENV_PASS': 'docker', 'CATALOGUE_DB_PORT_5432_TCP': 'tcp://172.17.0.2:5432'}
System check identified some issues:

WARNINGS:
catalogue.OpticalProduct.cloud_cover: (fields.W122) 'max_length' is ignored when used with IntegerField
    HINT: Remove 'max_length' from field
orders.OrderNotificationRecipients.classes: (fields.W340) null has no effect on ManyToManyField.
orders.OrderNotificationRecipients.satellite_instrument_group: (fields.W340) null has no effect on ManyToManyField.
search.Search.cloud_max: (fields.W122) 'max_length' is ignored when used with IntegerField
    HINT: Remove 'max_length' from field
search.Search.cloud_min: (fields.W122) 'max_length' is ignored when used with IntegerField
    HINT: Remove 'max_length' from field
search.Search.collection: (fields.W340) null has no effect on ManyToManyField.
search.Search.instrument_type: (fields.W340) null has no effect on ManyToManyField.
search.Search.license_type: (fields.W340) null has no effect on ManyToManyField.
search.Search.processing_level: (fields.W340) null has no effect on ManyToManyField.
search.Search.satellite: (fields.W340) null has no effect on ManyToManyField.
search.Search.spectral_group: (fields.W340) null has no effect on ManyToManyField.
Scanning folders in /home/web/django_project/data/landsat/
product folder: Landsat8 
search path: /home/web/django_project/data/landsat/Landsat8/*.txt 
xml_file: /home/web/django_project/data/landsat/Landsat8/L081750672017347JSA00_MTL.txt 
processing txt metadata
Processing txt files
Trying to update
Saving product and setting thumb
Product 1 updated.
jpeg_path: /home/web/django_project/data/landsat/Landsat8/L081750672017347JSA00_BQA.TIF
Thumbnails cannot be found
xml_file: /home/web/django_project/data/landsat/Landsat8/L081810702017341JSA00_MTL.txt 
processing txt metadata
Processing txt files
Trying to update
Not in catalogue: creating.
Product: L081810702017341JSA00
Saving product and setting thumb
Product 2 imported.
jpeg_path: /home/web/django_project/data/landsat/Landsat8/L081810702017341JSA00_BQA.TIF
Thumbnails cannot be found
xml_file: /home/web/django_project/data/landsat/Landsat8/L081810722017341JSA00_MTL.txt 
processing txt metadata
Processing txt files
Trying to update
Not in catalogue: creating.
Product: L081810722017341JSA00
Saving product and setting thumb
Product 3 imported.
jpeg_path: /home/web/django_project/data/landsat/Landsat8/L081810722017341JSA00_BQA.TIF
===============================
Products processed : 3 
Products updated : 1 
Products imported : 2 
Products failed to import : 0 
===============================

Then, I can search them from the browser: catalogue 525

The txt ingestor will look for the following fields in the metadata:

CORNER_UL_LAT_PRODUCT, CORNER_UL_LON_PRODUCT, CORNER_UR_LAT_PRODUCT, CORNER_UR_LON_PRODUCT, CORNER_LL_LAT_PRODUCT, CORNER_LL_LON_PRODUCT, CORNER_LR_LAT_PRODUCT, CORNER_LR_LON_PRODUCT, FILE_DATE, CLOUD_COVER, SENSOR_ID, UTM_ZONE

Maite-Machipi commented 6 years ago

Hi @boney-bun

i though i added my comment here already, it must have all been in my head. for this we will need to change the format of the thumbnail, i will be meeting with one of my colleagues so that he can show me how to generate the thumbnails, which will be used for the catalogue, i will inform you as soon as that is done.

lavhengwa commented 6 years ago

A critical item that must be done as priority # 1.

Maite-Machipi commented 6 years ago

hi @boney-bun

just a response for the above questions:

The ingestor should focus on the MTL
The name should be LO(letter o)8
The thumbnails have been generated the image ends with .Thumb.png
we dont have any xml at the moment the metadata come in .txt

just a quick question, will the ingestor ingest a folder like we do with Radarsat?

boney-bun commented 6 years ago

Hi @Maite-Machipi

I had ingested metadata in Landsat folder. The log is here: landsat ingestion log.txt

Note:

The ingestion searches for all subfolders in Landsat. the same functionality as in radarsat.
Don't forget to add -u parameter when doing ingestion to indicate that we prefer txt instead of xml.
The ingestion will fail on files such as: LC08_L1TP_170084_20180111_20180119_01_T1_MTL.txt. How do you want me to process this kind of files? The ingestor will look for files such as: LO81680762017346JSA00_MTL.txt. notice:
- there is not many underscores
- The failed metadata starts with LC. the ingestor expects LO.
the numbers at the end of the log are partially correct. I think this is related to the point number 2 above. Also, it's likely that there are a few data duplication spreading across different folders.

some latest Landsat metadata are in production now. please let me know if you find incorrect results.

boney-bun commented 6 years ago

Hi @Maite-Machipi

following up our last discussion, the ingestor can now recognize OLI/TIRS metadata (LC metadata) as shown in the below file log: landsatIngestionLog30Aug6PM.txt

i found interesting thing from the LC metadata in Test folder. the SENSOR_ID is OLI. hence, the ingestor will ingest the metadata, but it will classify the data as OLI. It should be 'OLI/TIRS'.

could you check again @Maite-Machipi ?

Maite-Machipi commented 6 years ago

Hi @boney-bun

i did some investigation on that, i was told that its because the file is missing band 10 and 11, so it only mentions OLI and not TIRS. therefore our internal solution is to try and rename the folders to LO that don't have Band 10 and Band 11. as soon as that is done i will test the LC, as long as the ingest can pick up LC then as soon as this issue is fixed on our side i can try and ingest the LC.

kartoza / catalogue

Landsat ingest #525