search

uktrade / mobius3

Continuously sync folder to S3, using inotify under the hood
MIT License
50 stars 3 forks source link

How to use mobius3 #43

Closed vinayKs21 closed 3 years ago

vinayKs21 commented 3 years ago

Hy is there any documentation of this tool regarding how to use it, I tried to run the python code you have mentioned in the REDME.md. It gives me the following error

SyntaxError: 'await' outside function

And when I try to use it with the cmd command, It gives me the following error

    raise KeyError(key) from None
KeyError: 'AWS_ACCESS_KEY_ID'

Where do I need to enter the key, please provide some documentation on how to use this tool, how can I enable the synchronization with this folder.

vinayKs21 commented 3 years ago

I have added my ACCESS KEY ID and SECRET ACCESS KEY into my environmental variables and when I run the following command:

mobius3 /home/vinay/Documents/VinayPersonal/Contraly/syncApp/someData/ vinay-codes https://s3.console.aws.amazon.com/ ap-south-1 --prefix s3://vinay-codes/experiment/

I got the following error

Traceback (most recent call last):
  File "/home/vinay/.local/bin/mobius3", line 8, in <module>
    sys.exit(main())
  File "/home/vinay/.local/lib/python3.8/site-packages/mobius3.py", line 1436, in main
    cleanup = loop.run_until_complete(async_main(syncer_args))
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/vinay/.local/lib/python3.8/site-packages/mobius3.py", line 1320, in async_main
    await start()
  File "/home/vinay/.local/lib/python3.8/site-packages/mobius3.py", line 414, in start
    await list_and_schedule_downloads(logger)
  File "/home/vinay/.local/lib/python3.8/site-packages/mobius3.py", line 994, in list_and_schedule_downloads
    path_etags = [
  File "/home/vinay/.local/lib/python3.8/site-packages/mobius3.py", line 994, in <listcomp>
    path_etags = [
  File "/home/vinay/.local/lib/python3.8/site-packages/mobius3.py", line 1266, in list_keys_relative_to_prefix
    token, keys_page = await list_first_page()
  File "/home/vinay/.local/lib/python3.8/site-packages/mobius3.py", line 1255, in list_first_page
    return await _list()
  File "/home/vinay/.local/lib/python3.8/site-packages/mobius3.py", line 1231, in _list
    raise Exception(code, body_bytes)
Exception: (b'302', b'<html>\r\n<head><title>302 Found</title></head>\r\n<body bgcolor="white">\r\n<center><h1>302 Found</h1></center>\r\n<hr><center>Server</center>\r\n</body>\r\n</html>\r\n')

It is not even specifying the error how to resolve this one, please help!

vinayKs21 commented 3 years ago

I forget to add {} between the url, So now i run the following command

mobius3 /home/vinay/Documents/VinayPersonal/Contraly/syncApp/someData vinay-codes https://{}.s3.console.aws.amazon.com/ ap-south-1 --prefix s3://vinay-codes/experiment/

Now I am getting the following error

[s3sync:start] [http:GET,https://vinay-codes.s3.console.aws.amazon.com/] [dns:vinay-codes.s3.console.aws.amazon.com,A] Nameserver failed: ('127.0.0.53', 53)
Traceback (most recent call last):
  File "/home/vinay/.local/lib/python3.8/site-packages/lowhaio.py", line 170, in request
    ip_addresses = (ipaddress.ip_address(parsed_url.hostname),)
  File "/usr/lib/python3.8/ipaddress.py", line 53, in ip_address
    raise ValueError('%r does not appear to be an IPv4 or IPv6 address' %
ValueError: 'vinay-codes.s3.console.aws.amazon.com' does not appear to be an IPv4 or IPv6 address

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vinay/.local/lib/python3.8/site-packages/lowhaio.py", line 173, in request
    ip_addresses = await dns_resolve(
  File "/home/vinay/.local/lib/python3.8/site-packages/aiodnsresolver.py", line 389, in resolve
    cname_rdata, qtype_rdata = await request_memoized(logger, fqdn, qtype)
  File "/home/vinay/.local/lib/python3.8/site-packages/aiodnsresolver.py", line 418, in request_memoized
    return await memoized_mutex()
  File "/home/vinay/.local/lib/python3.8/site-packages/aiodnsresolver.py", line 648, in runner
    result = await func(*args)
  File "/home/vinay/.local/lib/python3.8/site-packages/aiodnsresolver.py", line 421, in request_and_cache
    answers = await request_until_response(logger, fqdn, qtype)
  File "/home/vinay/.local/lib/python3.8/site-packages/aiodnsresolver.py", line 454, in request_until_response
    return await request_with_timeout(logger, timeout, addrs, fqdn, qtype)
  File "/home/vinay/.local/lib/python3.8/site-packages/aiodnsresolver.py", line 481, in request_with_timeout
    return await request(logger, addrs, fqdn, qtype, set_timeout_cause)
  File "/home/vinay/.local/lib/python3.8/site-packages/aiodnsresolver.py", line 583, in request
    raise DnsRecordDoesNotExist()
aiodnsresolver.DnsRecordDoesNotExist

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/vinay/.local/bin/mobius3", line 8, in <module>
    sys.exit(main())
  File "/home/vinay/.local/lib/python3.8/site-packages/mobius3.py", line 1436, in main
    cleanup = loop.run_until_complete(async_main(syncer_args))
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/vinay/.local/lib/python3.8/site-packages/mobius3.py", line 1320, in async_main
    await start()
  File "/home/vinay/.local/lib/python3.8/site-packages/mobius3.py", line 414, in start
    await list_and_schedule_downloads(logger)
  File "/home/vinay/.local/lib/python3.8/site-packages/mobius3.py", line 994, in list_and_schedule_downloads
    path_etags = [
  File "/home/vinay/.local/lib/python3.8/site-packages/mobius3.py", line 994, in <listcomp>
    path_etags = [
  File "/home/vinay/.local/lib/python3.8/site-packages/mobius3.py", line 1266, in list_keys_relative_to_prefix
    token, keys_page = await list_first_page()
  File "/home/vinay/.local/lib/python3.8/site-packages/mobius3.py", line 1255, in list_first_page
    return await _list()
  File "/home/vinay/.local/lib/python3.8/site-packages/mobius3.py", line 1228, in _list
    code, _, body = await signed_request(logger, b'GET', bucket_url, params=query)
  File "/home/vinay/.local/lib/python3.8/site-packages/mobius3.py", line 312, in _signed
    return await request(
  File "/home/vinay/.local/lib/python3.8/site-packages/lowhaio_retry.py", line 24, in _retry
    raise exception from exception.__cause__
  File "/home/vinay/.local/lib/python3.8/site-packages/lowhaio_retry.py", line 15, in _retry
    return await request(*args, **kwargs)
  File "/home/vinay/.local/lib/python3.8/site-packages/lowhaio.py", line 178, in request
    raise HttpDnsError() from exception
lowhaio.HttpDnsError
vinayKs21 commented 3 years ago

I guess there is some error in the specified URL, Now I tried with the following terminal command 

mobius3 /home/vinay/Documents/VinayPersonal/Contraly/syncApp/someData/ vinay-codes https://s3.console.aws.amazon.com/s3/buckets/{} ap-south-1 --prefix experiment/

and ends up with the following error

Traceback (most recent call last):
  File "/home/vinay/.local/bin/mobius3", line 8, in <module>
    sys.exit(main())
  File "/home/vinay/.local/lib/python3.8/site-packages/mobius3.py", line 1436, in main
    cleanup = loop.run_until_complete(async_main(syncer_args))
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/vinay/.local/lib/python3.8/site-packages/mobius3.py", line 1320, in async_main
    await start()
  File "/home/vinay/.local/lib/python3.8/site-packages/mobius3.py", line 414, in start
    await list_and_schedule_downloads(logger)
  File "/home/vinay/.local/lib/python3.8/site-packages/mobius3.py", line 994, in list_and_schedule_downloads
    path_etags = [
  File "/home/vinay/.local/lib/python3.8/site-packages/mobius3.py", line 994, in <listcomp>
    path_etags = [
  File "/home/vinay/.local/lib/python3.8/site-packages/mobius3.py", line 1266, in list_keys_relative_to_prefix
    token, keys_page = await list_first_page()
  File "/home/vinay/.local/lib/python3.8/site-packages/mobius3.py", line 1255, in list_first_page
    return await _list()
  File "/home/vinay/.local/lib/python3.8/site-packages/mobius3.py", line 1234, in _list
    root = ET.fromstring(body_bytes)
  File "/usr/lib/python3.8/xml/etree/ElementTree.py", line 1320, in XML
    parser.feed(text)
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 19, column 23
vinayKs21 commented 3 years ago

Could you please tell me what is the error with this command

 mobius3 /home/vinay/Documents/VinayPersonal/Contraly/syncApp/someData https://s3.console.aws.amazon.com/s3/buckets/vinay-codes/ https://s3.console.aws.amazon.com/s3/buckets/{}/ ap-south-1

I am still getting the below error 

xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 19, column 23

I have followed your --help and have done everything as shown in e.g but still not working. Help me.

vinayKs21 commented 3 years ago

Finally ! This command works on my terminal, Now looking forward to make the same thing run in the .py program.

mobius3 /home/vinay/Documents/VinayPersonal/Contraly/syncApp/someData vinay-codes https://{}.s3.ap-south-1.amazonaws.com/ ap-south-1 --prefix experiment/
vinayKs21 commented 3 years ago

Able to sync the local folder to the s3 bucket successfully.