singer-io / getting-started

This repository is a getting started guide to Singer.
https://singer.io
1.26k stars 148 forks source link

Save State #39

Closed btwq closed 6 years ago

btwq commented 6 years ago

In Step 5 the suggested Save State lines to execute:

› tap-fixerio --state state.json | target-gsheet -c config.json >> state.json › tail -1 state.json > state.json.tmp && mv state.json.tmp state.json (rinse and repeat)

I tried instead a single ">":

› tap-fixerio --state state.json | target-gsheet -c config.json > state.json

Logically this should simply overwrite an existing state file. Instead I encounter:

/var/www# tap-fixerio/bin/tap-fixerio --state state.fixerio.json | target-gsheet/bin/target-gsheet -c config.json > state.fixerio.json Traceback (most recent call last): File "tap-fixerio/bin/tap-fixerio", line 11, in sys.exit(main()) File "/var/www/tap-fixerio/lib/python3.5/site-packages/tap_fixerio.py", line 101, in main state = json.load(file) File "/usr/lib/python3.5/json/init.py", line 268, in load parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw) File "/usr/lib/python3.5/json/init.py", line 319, in loads return _default_decoder.decode(s) File "/usr/lib/python3.5/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Am I missing something here? It would be easier to have one line than the "rinse and repeat" method

cmerrick commented 6 years ago

> will wipe out the contents of the prior file before any of the processes start.

Verify with the following:

echo 'abc' > redirected.out
bash -c 'sleep 20 && echo def' > redirected.out 

Then in a different shell, immediately cat redirected.out - you'll see it's an empty file until the 20 seconds have elapsed, and then it contains def.