search

geonetwork / core-geonetwork

GeoNetwork is a catalog application to manage spatially referenced resources. It provides powerful metadata editing and search functions as well as an interactive web map viewer. It is currently used in numerous Spatial Data Infrastructure initiatives across the world.
http://geonetwork-opensource.org/
GNU General Public License v2.0
428 stars 489 forks source link

Can't get S3 support running in Docker #4466

Open MattReimer opened 4 years ago

MattReimer commented 4 years ago

Based on the docs here

and the #4248 It should now be possible to connect GeoNetwork to AWS S3 but I can't seem to do it.

I have a simple Dockerfile like so:

FROM geonetwork:postgres

COPY conf/config-spring-geonetwork.xml /usr/local/tomcat/webapps/geonetwork/WEB-INF/config-spring-geonetwork.xml

and the config-spring-geonetwork.xml specified has the following

  <bean id="s3credentials" class="org.fao.geonet.resources.S3Credentials">
   <!-- <property name="region" value="us-west-2"/>
   <property name="bucket" value="mybucket"/>
   <property name="keyPrefix" value="geonetwork"/> -->
   <!-- Only needed if you don't have a ~/.aws/credentials -->
   <!-- <property name="accessKey" value="MyAccessKey"/>
   <property name="secretKey" value="MySecretKey"/> -->
   <!-- Only needed when not using Amazon S3-->
   <!-- <property name="endpoint" value="sos-ch-dk-2.exo.io"/> -->
 </bean>
 <bean id="filesystemStore" class="org.fao.geonet.api.records.attachments.S3Store" />
 <bean id="resourceStore" class="org.fao.geonet.api.records.attachments.ResourceLoggerStore">
   <constructor-arg index="0" ref="filesystemStore"/>
 </bean>
 <bean id="resources" class="org.fao.geonet.resources.S3Resources"/>

According to the docs I can set all these values in environment variables:

    AWS_S3_PREFIX
    AWS_S3_BUCKET
    AWS_DEFAULT_REGION
    AWS_S3_ENDPOINT
    AWS_ACCESS_KEY_ID
    AWS_SECRET_ACCESS_KEY

And I've double-checked that they are being set but it's still creating the local data dir.

I'm not a Java / Tomcat expert so I'm hoping it's something I'm doing wrong.

gary-downforce commented 2 years ago

Exactly the same issue here. It complains if it can't access the S3 bucket, but when it can access the bucket it doesn't use it.

Also it doesn't seem to support using the Instance Profile on ECS, you have to supply an access key and secret.

FROM geonetwork:3.12-postgres

COPY conf/config-spring-geonetwork.xml /usr/local/tomcat/webapps/geonetwork/WEB-INF/config-spring-geonetwork.xml
ianwallen commented 1 year ago

Note that there is an implementation of JCLOUD that is not fully documented as it is still experimental. JCLOUD supports multiple cloud providers including AWS S3 (https://jclouds.apache.org/reference/providers/#blobstore-providers)

I was able to connect to Azure Blob storage during my testing however since I did not have access to S3 storage I did not test AWS however it should still be possible.

You can review the following for example connecting to Azure Blob storage.

https://github.com/geonetwork/core-geonetwork/pull/4570