search

cobbzilla / s3s3mirror

Mirror one S3 bucket to another S3 bucket, or to/from the local filesystem.
Other
426 stars 92 forks source link

Cannot copy from "subfolder" (prefix) to "root-folder" (without prefix) from src-bucket to dest-bucket #115

Open rwunsch opened 3 years ago

rwunsch commented 3 years ago

We are tying to copy an src-s3-bucket with a "prefix" (subfolder), to a des-s3-bucket without a "prefix" (subfolder)

Example: in the sec-bucket we have a subfolder "datastore": s3://src-bucket/datastore We would like to "copy" everything from the "s3://src-bucket/datastore" into "s3://target-bucket/datastore

Options we tried:

s3s3mirror -p "datastore" s3://src-bucket s3://target-bucket -> copies "datastore-folder" s3s3mirror -p "datastore" -d s3://src-bucket s3://target-bucket > fails with error s3s3mirror -p "datastore" -d "" s3://src-bucket s3://target-bucket -> copies "datastore-folder" s3s3mirror -p "datastore" -d " " s3://src-bucket s3://target-bucket -> creates " "-folder s3s3mirror -p "datastore" -d "." s3://src-bucket s3://target-bucket -> creates "."-folder s3s3mirror -p "datastore" -d . s3://src-bucket s3://target-bucket -> creates "."-folder s3s3mirror -p "datastore" -d / s3://src-bucket s3://target-bucket -> copies into "/"folder

How would one specify "using the s3-root" in the "-d" parameter?

cobbzilla commented 3 years ago

Did you try: s3s3mirror src-bucket/datastore target-bucket ?

rwunsch commented 3 years ago

Hi @cobbzilla ,

yes I tried that. I used a new git-clone of the repo, please find commands and test-results below. I might be doing something wrong - but do not know what :)

(Sidenote: it seems the log4j.xml is missing from "/target/classes/log4j.xml" in the repo - might me my miss as well, though).

Thanks for this cool and very helpful tool! And thanks for helping to support it!

Thanks and cheers - Robert

[12:15 wunsch@wunsch-WX-3 s3s3mirror] (master) > aws s3 ls s3://mdb-test-source --recursive
2021-02-09 10:21:22 1073741824 datastore/test123

[12:15 wunsch@wunsch-WX-3 s3s3mirror] (master) > aws s3 ls s3://mdb-test-target --recursive

[12:16 wunsch@wunsch-WX-3 s3s3mirror] (master) > ./s3s3mirror.sh -v s3://mdb-test-source/datastore s3://mdb-test-target
log4j:ERROR Could not parse url [file:target/classes/log4j.xml].
java.io.FileNotFoundException: target/classes/log4j.xml (No such file or directory)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at java.io.FileInputStream.<init>(FileInputStream.java:93)
        at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
        at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
        at org.apache.log4j.xml.DOMConfigurator$2.parse(DOMConfigurator.java:765)
        at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:871)
        at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:778)
        at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
        at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
        at org.slf4j.impl.Log4jLoggerFactory.<init>(Log4jLoggerFactory.java:66)
        at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:72)
        at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:45)
        at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
        at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
        at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:417)
        at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362)
        at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:388)
        at org.cobbzilla.s3s3mirror.MirrorMain.<clinit>(MirrorMain.java:23)
log4j:WARN No appenders could be found for logger (com.amazonaws.AmazonWebServiceClient).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

[12:18 wunsch@wunsch-WX-3 s3s3mirror] (master) > aws s3 ls s3://mdb-test-target --recursive
2021-02-10 12:17:40 1073741824 datastore/test123