Unable to start sinking without specifying start and end block

[let4be]

substreams-sink-postgres run -p "psql://substreams:let-me-in@localhost:5432/substreams?sslmode=disable" localhost:9000 substreams.yaml gives me errors like 2023-01-23T18:47:23.430+0400 ERRO (sink-postgres) substreams encountered an error {"error": "EOF"} do I always need to specify start and end block?... how is cursor that is saved in DB relates to this? because I also unable to continue syncing after termination when I DO specify start and end block... cursor is being ignored and I get no new at all from substream

What's making this worse this is somehow tied to substream production/development mode(currently non exposed, had to hack around the code). When you switch it substreams-sink-postgres behavior changes around start:end block and cursor

[maoueh]

Ok, there is a different going on in the issue, so let's try to keep things together in corresponding issue.

gives me errors like 2023-01-23T18:47:23.430+0400 ERRO (sink-postgres) substreams encountered an error {"error": "EOF"}

I don't think Development/Production has an impact here. I'm thinking it could be more a problem because maybe your chain is not fully synced.

do I always need to specify start and end block?

No it's not required, again could have something to do with your local instance.

because I also unable to continue syncing after termination when I DO specify start and end block... cursor is being ignored and I get no new at all from Substreams.

So let's focus on the EOF here in your issue for now. Try on StreamingFast endpoint for now. If you are testing on BSC, let me update BSC deployment to use latest version of everything. I'll tell you when done.

[let4be]

I don't think Development/Production has an impact here. I'm thinking it could be more a problem because maybe your chain is not fully synced.

Chain is in the process of syncing, but what does it have to do with this if Firehose can already stream blocks(100k+)? Having a fully synced chain is a very hefty requirement when I just want to check if things are working and how are they working.

EOF currently happens only in development mode(hacked substreams-sink-postgres which hardcoded production=true) when I do not specify start/end block. Production mode just doesn't consume anything at all doesn't matter what I do Basically default substreams-sink-postgres as it is - unusable(at the latest versions of everything). Don't know maybe I'm doing something wrong

[maoueh]

Chain is in the process of syncing, but what does it have to do with this if Firehose can already stream blocks(100k+)? Having a fully synced chain is a very hefty requirement when I just want to check if things are working and how are they working.

I'm not saying it's a requirement, I'm putting out hypothesis here.

We definitely not see this problem here, I tested https://github.com/streamingfast/substreams-sink-postgres#quickstart Run the sink command and let it sink (endpoint used was mainnet.eth.streamingfast.io:443) until ~4.9M before stopping it (I'm planning to let it run overnight to see where it lead me). This was in production mode, I also tested with new flag --development-mode, I was not able to reproduce neither. So far, never hit the error.

That's why I was putting the fact that maybe with your local deployment, something make it create "EOF" error which would mean we reach "end" of chain. The expected behavior when providing no block range is to sync from module's start block up to chain head than continue syncing with head of chain forever.

[let4be]

Got the token and tried running tutorial - everything works(eth mainnet)! I no longer get the EOF when starting without start:end block Even better - my own substream works as well(bsc mainnet)...

Although I tried adding -p for plaintext gRPC connection and the tool got silent - no sync/no errors...

[let4be]

I think this can be closed now, docs probably need to be updated to point out that right now this functionality(running without specifying a precise block range) is supported only when firehose is fully synced