Transfer service has a limit of 20 prefixes per transfer

iconara / bigshift

A tool for moving tables from Redshift to BigQuery

BSD 3-Clause "New" or "Revised" License

65 stars 10 forks source link

Transfer service has a limit of 20 prefixes per transfer #1

Closed iconara closed 8 years ago

iconara commented 8 years ago

BigShift's strategy of transferring exactly the files in the unload manifest seems to have backfired. When an unload results in more than 20 files the transfer will error. The reason is that you're only allowed to specify 20 include prefixes in a transfer.

iconara commented 8 years ago

Fixed in f486c3e90bcf57a3b31b978fe2cfdad67faaa0ac/#3