search

adamchainz / dynamodb_utils

A toolchain for Amazon's DynamoDB to make common operations (backup, restore backups) easier.
ISC License
12 stars 1 forks source link

Add modular IO and Multiprocess back-ends #14

Closed brandond closed 5 years ago

brandond commented 7 years ago

Takes care of #6

brandond commented 7 years ago

I also changed the prints to logging calls, and replaced all the explicit exit calls with exceptions that are now caught by the CLI. Should make it easier to call from other code without having it just up and quit on you if something goes wrong.

brandond commented 7 years ago

After adding S3 support, I tried wrapping a basic handler around the dumper and running it in Lambda, but it turns out the Lambda environment is missing /dev/shm, which breaks the semaphore locking that multiprocessing.Pool requires... so I had to refactor that sooner than I'd planned.

The new lambda multiprocessing back-end will do an async invoke of itself to do the work of backing up segments or keys. Lambda does some funny things with module loading that make it so you can't just point the handler directly at a module within an installed package, so it needs a simple wrapper like this in function.py:

from dynamodb_utils.awslambda import handler

With that in place, you can set the handler to function.handler and everything is hunky-dory.

That probably bears documenting somewhere if this is going to be merged.

adamchainz commented 7 years ago

Sorry I haven't got the time to look at all the implementation at once. This is quite a big PR since it's also adding the Lamdba invocation stuff. Also you should also be able to ship the wrapper function as well as a separate one-file module called something like dynamodb_dumper_lambda.py

brandond commented 7 years ago

The problem with shipping a wrapper is that the Lambda importer doesn't work right - if you specify package.module.function as the handler it will search PYTHONPATH for package.module.py instead of package/module.py or package/module/__init__.py which then breaks all the relative imports as well because the module name is mangled by the included path components.

I suppose that sticking it in a separate package and then explicitly importing dynamodb_utils from there might work. Would also have to document that the handler needs to be referenced as package/module.function.

adamchainz commented 7 years ago

@brandond we don't even need a package, we can just have a module

brandond commented 7 years ago

Not sure how setuptools handles modules without a package - have to play with it.

brandond commented 5 years ago

AWS has had support for native ddb backups for a while now.

adamchainz commented 5 years ago

Yes it has. I might even retire this package as I haven't used DynamoDB in years. Thanks for your work though!