Catalogue's thumbnail missing

[rischanlab]

Problem

• Some sensors do not have thumbnail

  Solution

• Make a sensor list which does not have thumbnail
• Send email to @kmithi asking about thumbnail
• [ ] Ask @kmithi to copy over all existing thumbnails from sac-live server to test server
• [ ] SPOT1
• [ ] SPOT2
• [ ] SPOT4
• [ ] SPOT5
• [ ] SPOT6
• [ ] SPOT7
• [ ] CBERS 2B CCD
• [x] CBERS MUX
• [x] CBERS WFI
• [x] CBERS P10
• [x] CBERS P5M
• [ ] Lansat5
• [ ] Landsat7
• [ ] Landsat8

[rischanlab]

Hi guys. I saw thumbs in the catalogue live server but It seems I could not load all files to the docker container. This is the ls of directory:

Local directory:

root@testenv:/EOSHARE/Karabo/CATALOGUE/thumbnails_cat3# ls
AQA  cache  E2  L3  L5  N11  N14  N16  N9   RE2  RE4  S1  S3  S5    S-C  ZA2
C2B  E1     L2  L4  L7  N12  N15  N17  RE1  RE3  RE5  S2  S4  SACC  TER
root@testenv:/EOSHARE/Karabo/CATALOGUE/thumbnails_cat3# 

Docker container:

root@uwsgi:/home/web/django_project/thumbs_out# ls
CB04  S1  S2  S4  S5  cache

I've tried to check the size of thumbnails_cat3 directory but too much time it takes and giving not responding. The docker container size:

root@testenv:/home/web/catalogue/deployment# docker ps -s
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS                                      NAMES                     SIZE
f1380ac2e1f1        catalogue_uwsgi           "python manage.py mig"   5 minutes ago       Up 5 minutes        8080/tcp                                   catalogue_uwsgi_run_119   283.4 kB (virtual 2.468 GB)
c18553c5141b        nginx                     "nginx -g 'daemon off"   5 minutes ago       Up 5 minutes        80/tcp, 443/tcp, 0.0.0.0:62080->8080/tcp   catalogue-web             2 B (virtual 190.5 MB)
0bb98b3f4500        catalogue_uwsgi           "uwsgi --ini /uwsgi.c"   5 minutes ago       Up 5 minutes        8080/tcp                                   catalogue-uwsgi           252.7 kB (virtual 2.468 GB)
b9166d6cfa8d        catalogue_worker          "celery worker -A cat"   8 days ago          Up 8 days           8080/tcp                                   catalogue_worker_1        373.2 kB (virtual 2.468 GB)
bd65f624cded        library/rabbitmq          "/docker-entrypoint.s"   8 days ago          Up 8 days           4369/tcp, 5671-5672/tcp, 25672/tcp         catalogue_rabbitmq_1      0 B (virtual 300.8 MB)
07f212b09ef8        kartoza/pg-backup:9.4     "/start.sh"              12 weeks ago        Up 4 weeks          5432/tcp                                   catalogue-db-backups      1.071 MB (virtual 497.9 MB)
d32576b47c75        catatnight/postfix        "/bin/sh -c '/opt/ins"   12 weeks ago        Up 4 weeks                                                     catalogue-smtp            11.42 MB (virtual 268 MB)
6404c753dbf8        kartoza/postgis:9.4-2.1   "/bin/sh -c /start-po"   12 weeks ago        Up 6 days           5432/tcp                                   catalogue-db              3.915 GB (virtual 4.411 GB)

df -h

root@uwsgi:/home/web/django_project# df -h
Filesystem                                                           Size  Used Avail Use% Mounted on
none                                                                  73G   73G     0 100% /
tmpfs                                                                2.0G     0  2.0G   0% /dev
tmpfs                                                                2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/disk/by-uuid/d02bdc6c-2230-40ce-8407-079095f72488                73G   73G     0 100% /var/log
41.74.147.98:/lustre/lfs01/EOSHARE/Karabo/CATALOGUE/thumbnails_cat3  151T  134T  9.8T  94% /django_project/thumbs_out
shm                                                                   64M     0   64M   0% /dev/shm

[rischanlab]

Hi @gubuntu @lucernae said that there is no maximum size for the docker container. Could you give me a suggestion @lucernae ?

[lucernae]

@rischanlab As I see it, I think this is because the container itself is full. In above stats we see that:

none                                                                  73G   73G     0 100% /

This is the root partition of the container, and it is full. It means the container itself is already full. maybe there are many temporary process inside the container that is using disk space of the container. For example, in other project I usually used temporary file inside /tmp folder (for QGIS analysis stuffs), because it will build up, I mount host file system to /tmp folder so the file itself is not saved inside the container, but on the host filesystem. You can probably check what is the cause of file build up in your container, perhaps you used temp file, or caching some big logs in a file, but not mount it on host filesystem? After you identified it, I suspect it is not important files, you can just delete the whole container using docker rm, then rebuild it again to get a fresh container (with extra new space). Then properly mount troublesome folders to host filesystem, so the file will build up in the host, not in the container.

For the thumbnail row:

41.74.147.98:/lustre/lfs01/EOSHARE/Karabo/CATALOGUE/thumbnails_cat3  151T  134T  9.8T  94% /django_project/thumbs_out

I don't see any problem with it. Sure it is a big one, 134T :v. I wonder what that folders contains. But it still has excess 9.8T which is pretty big. So I'm sure this is not the problem. Just make sure there were no unnecessary files in there to save some space. AFAIK, there were no limit on how big the host filesystem size is. For example, 151T here is the capacity of the server disk (I suppose). It can be resized from the host filesystem. Meanwhile, the size of the docker container here: (73G) is not that dynamic. It is the size of the container itself. It is different by the hardware you used. For example, in mac I can resize it because the docker container is a virtual box disk (It can be different in linux). So the size itself has no maximum, but it needs to be defined when creating the container. Since your container already full, best thing I can suggests is to delete the container and rebuild. Full container means possibly you can't do usual command stuff inside the container, like ls or df -h, since the /tmp directory is full and the container can't store temporary calculations. :p

[lucernae]

Just to confirm. The solution I proposed is to solve the 'full container' issue. However, I don't know does 134T size of the thumbnail is sane or not. It is pretty big, though. Ideally you don't want to make the host itself full. For example, even if I mount container's /tmp folder to host filesystem. I created a script which deleted the files regularly, so it doesn't build up over time unnecessarily.

[gubuntu]

Also we should be mounting just a directory or directories in the Gluster as volume(s) in the container and not the whole device.

[cchristelis]

@rischanlab, you are looking at the incorrect directory:

/home/web/catalogue/django_project/thums_out/ then you will find second directory.

try adding the directory

/home/web/thumbs/ in the compose file.

[rischanlab]

Hi @cchristelis I just following this, because in the setting Thumbs_root is on that path. I could not do it now. I am not able to access catalogue test server.

but I will try that when I able to login again.

Last login: Mon Jun 20 09:23:56 on ttys000
[rischan@mbp ~]$ ssh catalogue 
ssh: connect to host 41.74.158.3 port 22: Operation timed out
[rischan@mbp ~]$ ssh catalogue 
ssh: connect to host 41.74.158.3 port 22: Operation timed out
[rischan@mbp ~]$ ssh catalogue 
^C
[rischan@mbp ~]$ ping 41.74.158.4
PING 41.74.158.4 (41.74.158.4): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3
Request timeout for icmp_seq 4
^C
--- 41.74.158.4 ping statistics ---
6 packets transmitted, 0 packets received, 100.0% packet loss
[rischan@mbp ~]$ ping 41.74.158.3
PING 41.74.158.3 (41.74.158.3): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3
Request timeout for icmp_seq 4
Request timeout for icmp_seq 5
^C
--- 41.74.158.3 ping statistics ---
7 packets transmitted, 0 packets received, 100.0% packet loss

[rischanlab]

Hi @cchristelis @gubuntu I've mount /home/web/thumbs to docker container. Previously, I was wrong I mount /EOSHARE/Karabo/CATALOGUE/thumbnails_cat3 and @cchristelis just told me.

I am still checking with searching, whether all thumbs are loaded on the web or not.

Thanks.

[rischanlab]

Hi @cchristelis I have a question.

Is all thumbs files already copy to thumbs folder?

Most of folders inside thumbs directory are empty, total size of that folder is 1,3 G. Its so difference with /EOSHARE/Karabo/CATALOGUE/thumbnails_cat3 which has size 151T.

root@testenv:/home/web# du -h thumbs
4,0K    thumbs/S4/2000/12/30
8,0K    thumbs/S4/2000/12
...
4,0K    thumbs/N16/2001/8/4
4,0K    thumbs/N16/2001/8/7
4,0K    thumbs/N16/2001/8/16
4,0K    thumbs/N16/2001/8/1
4,0K    thumbs/N16/2001/8/15
4,0K    thumbs/N16/2001/8/6
4,0K    thumbs/N16/2001/8/17
4,0K    thumbs/N16/2001/8/27
4,0K    thumbs/N16/2001/8/21
4,0K    thumbs/N16/2001/8/3
4,0K    thumbs/N16/2001/8/5
4,0K    thumbs/N16/2001/8/20
128K    thumbs/N16/2001/8
4,0K    thumbs/N16/2001/11/12
4,0K    thumbs/N16/2001/11/26
4,0K    thumbs/N16/2001/11/24
4,0K    thumbs/N16/2001/11/2
4,0K    thumbs/N16/2001/11/28
4,0K    thumbs/N16/2001/11/30
4,0K    thumbs/N16/2001/11/14
4,0K    thumbs/N16/2001/11/18
4,0K    thumbs/N16/2001/11/9
4,0K    thumbs/N16/2001/11/10
4,0K    thumbs/N16/2001/11/25
4,0K    thumbs/N16/2001/11/22
4,0K    thumbs/N16/2001/11/23
4,0K    thumbs/N16/2001/11/29
4,0K    thumbs/N16/2001/11/8
4,0K    thumbs/N16/2001/11/11
4,0K    thumbs/N16/2001/11/13
4,0K    thumbs/N16/2001/11/19
4,0K    thumbs/N16/2001/11/4
...
2,6M    thumbs/AQA/2001/3
496K    thumbs/AQA/2001/5/9
504K    thumbs/AQA/2001/5/25
492K    thumbs/AQA/2001/5/1
512K    thumbs/AQA/2001/5/17
2,0M    thumbs/AQA/2001/5
26M thumbs/AQA/2001
89M thumbs/AQA
1,3G    thumbs

[cchristelis]

@rischanlab, either way we should use a mounted directory. @kmithi, please advise. Are we going to be using EOSHARE or another mounted directory?

[kmithi]

Hi, just to be that I understand your question, do you want to move away from using the EOSHARE and remount a different directory? Both cases are possible, please let me know your preference.

Please also note that my access to emails will be limited as I will be involved in a systems upgrade at HBK with our service provider for this week, with this note please keep Maite in copy of all communication especially that which requires urgent attention.

[rischanlab]

Hi @kmithi Ok please could you remount again EOSHARE? I will try one more time, if that is not possible to mount all thumbs from EOSHARE I'll find another way.

How do you think @cchristelis ?

Thanks.

[kmithi]

Hi Rischan,

How are you accessing the EOSHARE? Please user the following credentials:

Username / Password = eoshare / eoshare

Regards

From: Rischan Mafrur [mailto:notifications@github.com] Sent: 21 June 2016 09:11 AM To: kartoza/catalogue Cc: Karabo Mithi; Mention Subject: [MASSMAIL]Re: [kartoza/catalogue] Catalogue's thumbnail missing (#435)

Hi @kmithihttps://github.com/kmithi Ok please could you remount again EOSHARE? I will try one more time, if that is not possible to mount all thumbs in the EOSHARE I'll find another way.

How do you think @cchristelishttps://github.com/cchristelis ?

Thanks.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/kartoza/catalogue/issues/435#issuecomment-227358565, or mute the threadhttps://github.com/notifications/unsubscribe/AK-5qg-quGj8OVHKTWx7_Wr3wc-gcY6tks5qN46egaJpZM4ItKdV.

Karabo Mithi | Systems Administrator SANSA Earth Observation

T: +27 12 844 0382 | F: +27 12 844 0396 | Web: http://www.sansa.org.za Enterprise Building, Mark Shuttleworth Street, Innovation Hub, Pretoria 0087 | PO Box 484, Silverton, 0127 [SANSA Logo]

This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Only authorised signatories are competent to enter into agreements on behalf of the South African National Space Agency (SANSA) and recipients are thus advised that the content of this message may not be legally binding on SANSA and may contain the personal views and opinions of the author, which are not necessarily the views and opinions of SANSA. If you have received this communication in error, you must neither take any action based upon its contents, nor copy or show it to anyone and we advise that you please notify us immediately and destroy the original message.

[rischanlab]

Hi @kmithi

Could you explain me more, I did not see any files inside /EOSHARE how to use that credentials? do you know this one @cchristelis ?

Thanks

[kmithi]

May you please log out so that I can have a look the mount?

[rischanlab]

Hi @kmithi I've logged out and login again. and this is when I ls -al

[kmithi]

Can you log out completely? I will advise as to when you may re log in again.

[rischanlab]

I did @kmithi

Please could you tell me if that files available?

Thanks

[kmithi]

I am urgently required to assist with our terminal upgrade at HBK as mentioned yesterday, I will hand you over to my colleague at Pretoria (CC’d) to assist.

@tramokgopa

Could you please verify the NFS mount on the testenv for the EOSHARE. I will follow up with you around 14H00 when I get the chance, call me if need be as my email access will be limited.

[gubuntu]

gavin@testenv:~⟫ sudo mount /EOSHARE
[sudo] password for gavin: 
mount.nfs: Connection timed out

from fstab:

41.74.147.98:/lustre/lfs01/EOSHARE /EOSHARE     nfs     defaults        0       0

eoshare host is up so why is nfs timing out?

gavin@testenv:/home/web/catalogue⟫ ping 41.74.147.98
PING 41.74.147.98 (41.74.147.98) 56(84) bytes of data.
64 bytes from 41.74.147.98: icmp_seq=1 ttl=63 time=1.30 ms

[kmithi]

@kmithi

Please send me the log in details for the server so that I can connect.

[gubuntu]

@tramokgopa these are my login details:

Host catalog_test
  User gavin
  Port 22
  HostName 41.74.158.3

[kmithi]

@tramokgopa I have a window opened which I’ll use to continue where I left off. I’ll keep you posted on whether you may continue or not.

[rischanlab]

Hi @kmithi

OK let's check out this. I've search image sensor in old catalogue (sac live) http://catalogue.sansa.org.za/

And there is the results: All thumbs are not available. Some of them has icon no thumbs and some of them it seems has thumbnail but not loaded correctly.

SPOT1

SPOT 2

SPOT2 with difference page.

SPOT4

SPOT 5

SPOT5 with difference page.

CB2

L5

L7

So I just confused, most thumbnails on the sac live are not available. I created ingestor script for L8, S6, S7, CBERS04 and only CBERS04 which has thumbnails and I've ingested the thumbnails.

@kmithi Could you please confirm the sensors which has thumbnails?

cc @gubuntu @cchristelis

[kmithi]

@rischanlab

I will get back to you on the thumbnails.

In the meantime may you please verify access to the EOSHARE, the problem should be resolved, please let me know if it now accessible from your side?

[kmithi]

@rischanlab

Please note the following discoveries on the new catalogue:

1.  The close button does not work, clicking anywhere outside of the box closes the popup. Please see catalogue_01.jpg.

2.  It seems as if some images are strips and not scenes, ideally customers should only be able to order scenes and not the complete data strip. Please see catalogue_02.jpg.

[rischanlab]

Ok Thanks @kmithi I am able to access it now. So, finally the problem, previously the docker container is out of space. Now I can mount all thumbs to docker container.

I am still testing using search menu to see the thumb show up or not. Anyway, in the sac_live all thumbs does not show up, could you confirm that @kmithi as the screenshoot that I shere before?

[rischanlab]

Hi @kmithi I did not see the pictures which do you mean, catalogue_01.jpg catalogue_02.jpg? where did you upload those pictures?

Thanks.

[kmithi]

@rischanlab

Please find attached.

[gubuntu]

@kmithi please move these two items into two new issues and I don't believe they are blockers to going live.

regarding this Issue I propose we close it and proceed to go live, then create a new Issue for finding / generating / populating missing thumbnails as part of the next phase.

[kmithi]

OK. That will be fine.

[rischanlab]

Hi @kmithi about your problems. I have made a separated issue which are #443 and #444 could you check it?.

Thanks.