Import books into development environment

I've made some changes to the Book Tracker's configuration in order to make this more straightforward. After pulling the latest code on develop, remove the following keys from your config/development.yml and test.yml files:

aws_endpoint
aws_access_key_id
aws_secret_access_key
s3_endpoint
book_bucket
book_key_prefix
temp_bucket

Add the following section to development.yml:

storage:
  books:
    region:            us-east-2
    endpoint:          
    bucket:            medusa-main
    key_prefix:        813/1127
    access_key_id:     
    secret_access_key: 
  temp:
    region:            us-east-1
    endpoint:          http://localhost:9000
    bucket:            book-tracker-temp
    access_key_id:     minioadmin
    secret_access_key: minioadmin

Add the following section to test.yml:

storage:
  books:
    region:            us-east-1
    endpoint:          http://localhost:9000
    bucket:            book-tracker-books-test
    key_prefix:
    access_key_id:     minioadmin
    secret_access_key: minioadmin
  temp:
    region:            us-east-1
    endpoint:          http://localhost:9000
    bucket:            book-tracker-temp-test
    access_key_id:     minioadmin
    secret_access_key: minioadmin

Then you should be able to do:

aws login
bin/rails books:import

When I just tried this, my import went on for a while and then failed with "Import failed: The provided token has expired." That's an issue with the UofI aws login command not rotating its credentials properly (that's why we have to keep doing aws login periodically when we use it) but you should still end up with at least some books in your database. It's not important to have all of them. Maybe it would work to do an aws login every 45 minutes or whatever while the import is running if you want to import all 800,000+ books.

medusa-project / book-tracker

Import books into development environment #32