Update parse start script to handle the case that fsimage txid is cov…

fengnanli commented 5 years ago

Sometimes the edit logs doesn't end exactly with the txid from fsimage, so we need to find the proper range which covers this id and thus to pick the right edit log

fengnanli commented 5 years ago

@xkrogen Please review this one and thanks in advance!

xkrogen commented 5 years ago

Hey @fengnanli , thanks for reporting this! I am curious, when will it occur that the edits file does not match the fsimage txn ID? I was under the impression that it always will; I've never seen it differ on our cluster. Maybe there are changes in newer versions of Hadoop which cause this?

The code looks good. I have two minor comments:

If we already use awk for processing, can we do away with the cut and simply do awk -F'-' to grab only the ending txn ID?
When using awk, I'm used to seeing the single quotes appear outside of the curly braces like '{ if .... }'. Is there any difference with the way you have specified it? I actually was under the impression that they way you specified it would not work because bash would interpret the curly braces as special characters. I should note that I am by no means a bash expert, though.

fengnanli commented 5 years ago

@xkrogen In our cluster it seems there is never an edit log ending with the fsimage id. I am not sure why, and we are using 2.8.2. Comments applied. I am not bash expert either :)

linkedin / dynamometer

Update parse start script to handle the case that fsimage txid is cov… #78