search

GFDRR / geonode

GFDRR Lab GeoNode
https://www.geonode-gfdrrlab.org
GNU General Public License v3.0
2 stars 3 forks source link

batch upload to geonode #34

Closed stufraser1 closed 7 years ago

stufraser1 commented 7 years ago

@Vampouille @fvanderbiest Please can you confirm whether the new Geonode has full functionality to enable batch upload of layers?

On http://45.55.174.20/ we would copy files from local to a tmp folder on server, then import a batch using: cd /var/www/geonode/wsgi/geonode python manage.py importlayers Has this been replicated on 54.229.171.111/ ?

Please also confirm that files/folders in _54.229.171.111//mnt/geoservergeodata/hazard are not required, once files have been uploaded to _/mnt/apache_nasdata/uploaded I believe this is the case.

fvanderbiest commented 7 years ago

The importlayers command is standard geonode feature. As such, it is available in the new setup.

Devs at camptocamp can run the command, simply sshing into the django container : 

root@0f0f6a050554:/usr/src/app# python manage.py help

Type 'manage.py help <subcommand>' for help on a specific subcommand.

Available subcommands:

[account]
    expunge_deleted

[auth]
    changepassword
    createsuperuser

[avatar]
    rebuild_avatars

[base]
    backup
    fixsitename
    helpers
    migrate_layers
    restore

[django]
    check
    compilemessages
    createcachetable
    dbshell
    diffsettings
    dumpdata
    flush
    makemessages
    makemigrations
    migrate
    runfcgi
    shell
    showmigrations
    sql
    sqlall
    sqlclear
    sqlcustom
    sqldropindexes
    sqlflush
    sqlindexes
    sqlmigrate
    sqlsequencereset
    squashmigrations
    startapp
    startproject
    syncdb
    test
    testserver
    validate

[django_extensions]
    admin_generator
    clean_pyc
    clear_cache
    compile_pyc
    create_app
    create_command
    create_jobs
    create_template_tags
    describe_form
    drop_test_database
    dumpscript
    export_emails
    find_template
    generate_secret_key
    graph_models
    mail_debug
    notes
    passwd
    pipchecker
    print_settings
    print_user_for_session
    reset_db
    runjob
    runjobs
    runprofileserver
    runscript
    runserver_plus
    set_default_site
    set_fake_emails
    set_fake_passwords
    shell_plus
    show_template_tags
    show_templatetags
    show_urls
    sqlcreate
    sqldiff
    sqldsn
    sync_s3
    syncdata
    unreferenced_files
    update_permissions
    validate_templates

[djcelery]
    celery
    celerybeat
    celerycam
    celeryd
    celeryd_detach
    celeryd_multi
    celerymon
    djcelerymon

[geoserver]
    updatelayers

[gis]
    inspectdb
    ogrinspect

[guardian]
    clean_orphan_obj_perms

[layers]
    importlayers

[maps]
    changemaplayertitle
    updatemaplayerip

[modeltranslation]
    loaddata
    sync_translation_fields
    update_translation_fields

[services]
    importservice
    testservices

[sessions]
    clearsessions

[sitemaps]
    ping_google

[staticfiles]
    collectstatic
    findstatic
    runserver

[tastypie]
    backfill_api_keys

If you have to do it by yourself, we have to upgrade your account so you can SFTP and SSH to the server.

stufraser1 commented 7 years ago

It would be useful to have access to do this ourselves, so we can upload data without having to request this from camptocamp. Please provide access, if possible.

fvanderbiest commented 7 years ago

No problem, doing it asap.

Vampouille commented 7 years ago

@stufraser1 I have just setup your ssh access, so you should be able to connect with the following configuration :

You have to upload files in /mnt/apache_nas_data/tmp/ using sftp protocol and then connect through ssh and finally run python manage.py ....

Please let us know if you run into any issue.

stufraser1 commented 7 years ago

I'm unable to connect - have you added my IP address 2.30.124.184 to this - that had to be added for me to connect to 54.229.171.111:2222 Thanks

fvanderbiest commented 7 years ago

@saimonn can you check the security groups for Stuart to access int.geonode-gfdrrlab.org using ports 2222 and 2223 from 2.30.124.184 ?

Vampouille commented 7 years ago

Related internal issue: https://jira.camptocamp.com/browse/INF-159

saimonn commented 7 years ago

Les ports 2222,2223 et 5432 devraient maintenant être ouverts sur int + prod

saimonn commented 7 years ago

Oops, forgot the answer in English; TCP ports 2222, 2223 and 5432 should now be open on integration and production instances from 2.30.124.184

Sorry for the delay.

stufraser1 commented 7 years ago

@saimonn @fvanderbiest Thank you - I have been able to copy files via sftp to /mnt/apache_nas_data/tmp/

However, in trying to run _python manage.py importlayers /mnt/apache/nasdata/tmp/PHP100.tif I encounter the error _geonode.GeoNodeException: Please pass a filename or a directory name as the "incoming" parameter, instead of /root/mnt/apache/nasdata/tmp/PHP100.tif: <type 'str'>

This appears to have occurred before: https://github.com/GeoNode/geonode/issues/1759 but may be unrelated -

@fvanderbiest Do you experience the same issue if trying to upload that file?

ErikKBethke commented 7 years ago

Hi @saimonn, @Vampouille, and @fvanderbiest,

Would it be possible to open these ports to my IP address as well? I cannot connect to int.geonode-gfdrrlab.org (54.246.242.223:2223). If there is something that I am still missing beyond this, please let me know.

IP: 2601:146:c900:8d90:109a:6a8e:c48e:c2a2

fvanderbiest commented 7 years ago

IP: 2601:146:c900:8d90:109a:6a8e:c48e:c2a2

Not sure we are able to handle IPv6 addresses... I'm gonna ask our sysadmins. In the mean time, are you sure no IPv4 address is available ?

fvanderbiest commented 7 years ago

However, in trying to run python manage.py importlayers /mnt/apache/nas_data/tmp/PHP100.tif I encounter the error geonode.GeoNodeException: Please pass a filename or a directory name as the "incoming" parameter, instead of /root/mnt/apache/nas_data/tmp/PHP100.tif: <type 'str'>

Sorry for the late answer Stuart. I cannot find /root/mnt/apache_nas_data/tmp/PHP100.tif anymore. Is this still an issue ?

ErikKBethke commented 7 years ago

Hi @fvanderbiest,

Sorry, my IPv4 address is available: 192.168.0.13

Thanks!

fvanderbiest commented 7 years ago

This one is your internal (private) IP. To get your public IP, you can go to https://www.whatismyip.com/

stufraser1 commented 7 years ago

@fvanderbiest Ref 'I cannot find /root/mnt/apache_nas_data/tmp/PHP100.tif anymore. Is this still an issue ?' At that time I was working on int.geonode-gfdrrlab.org It doesn't appear on www.geonode-gfdrrlab.org as I've not uploaded there. Can you copy all files directly from int.geonode-/*/tmp to the www.geonode- server (to a new tmp folder)? Then I can test the batch upload.

fvanderbiest commented 7 years ago

Can you copy all files directly from int.geonode-/*/tmp to the www.geonode- server (to a new tmp folder)?

Sure. Copying ...

fvanderbiest commented 7 years ago

All done.

stufraser1 commented 7 years ago

Thank you - confirmed all data copied to www.geonode server, and I have deleted the tmp folder from int server. I have been able to upload data using /usr/src/app# python manage.py importlayers --user=stu /mnt/apache_nas_data/tmp/<filename>.tif via Putty. All sorted on batch upload side for me.

All that remains in this issue is to enable @ErikKBethke access (once he has provided IP) and ensure he can do the same.

ErikKBethke commented 7 years ago

Sorry @fvanderbiest, my public IPv4 is: 73.172.216.47

fvanderbiest commented 7 years ago

Sorry for the delay Erik, I'm doing my best to open your access ASAP.

fvanderbiest commented 7 years ago

TCP ports 2222, 2223 and 5432 should now be open on integration and production instances from 2.30.124.184

@saimonn can you configure the same accesses for IP 73.172.216.47 ? Thanks !

saimonn commented 7 years ago

that's open for 73.172.216.47

fvanderbiest commented 7 years ago

@ErikKBethke can you try again ? Thanks.

ErikKBethke commented 7 years ago

@fvanderbiest I am now able to connect to both instances, thank you!

ErikKBethke commented 7 years ago

However, in trying to run python manage.py importlayers /mnt/apache/nas_data/tmp/PHP100.tif I encounter the error geonode.GeoNodeException: Please pass a filename or a directory name as the "incoming" parameter, instead of /root/mnt/apache/nas_data/tmp/PHP100.tif: <type 'str'>

Following up, I am encountering difficulties similar to the ones that Stu found and posted in an earlier comment. There are several files in the tmp folder, but this exception is popping up when attempting either a batch or single import.

stufraser1 commented 7 years ago

@ErikKBethke was this resolved for you?

fvanderbiest commented 7 years ago

Tested fully functional on int today, closing.

root@7dce90566d12:/usr/src/app# python manage.py importlayers --user=jacroute /usr/local/lib/python2.7/site-packages/gisdata/data/good/raster/test_grid.tif

[created] Layer for '/usr/local/lib/python2.7/site-packages/gisdata/data/good/raster/test_grid.tif' (1/1)

Finished processing 1 layers in 7.0 seconds.

1 Created layers
0 Updated layers
0 Skipped layers
0 Failed layers
7.000000 seconds per layer

Please reopen if required.

fvanderbiest commented 7 years ago

BTW, side note for my colleague @Vampouille I stumbled upon this WARNING while uploading: https://github.com/GeoNode/geonode/pull/2711

Fixed by migrating the TEMPLATE_* settings (from local_settings.py on int and prod) into a single TEMPLATES var as per https://github.com/GeoNode/geonode/pull/2711/commits/454855a9a523727bc9248eb715f9f68d9f4fc385#diff-244cdfc4e5e50d8ee9386be986629b6aR381