search

treasure-data / digdag

Workload Automation System
https://www.digdag.io/
Apache License 2.0
1.3k stars 221 forks source link

Fix query layout generated by td:create_table parameter #17

Closed xerial closed 8 years ago

xerial commented 8 years ago

Queries with magic comment like this:

-- @TD reducers: 4
SELECT ...

will be replaced as: 

INSERT OVERWRITE TABLE ...
-- @TD reducers: 4
SELECT ...

Then a magic comment in Hive becomes ineffective. And also this can be problematic if a customer embeds meaningful message in the first line comment.

A possible solution for this problem would be skipping the first comment lines, then embedding INSERT OVERWRITE statement.

frsyuki commented 8 years ago

This is solved by 8dc39fd20bdae5e2da53f9e8f1be311e616a1104. Now digdag adds INSERT or CREATE after all comment lines.