klbostee / dumbo

Python module that allows one to easily write and run Hadoop programs.
http://projects.dumbotics.com/dumbo
1.04k stars 146 forks source link

pre-outputs not deleted automatically anymore #36

Closed klbostee closed 13 years ago

klbostee commented 13 years ago

Pre-outputs aren't deleted automatically anymore since we introduced the DAG capabilities for job flows. The reason for this is that

backend = get_backend(opts)
fs = backend.create_filesystem(opts)

gets executed at a point where 'opts' doesn't contain the 'hadoop' option anymore (since this option gets removed as part of running the iteration).