search

indexsupply / shovel

An Ethereum to Postgres indexer
https://indexsupply.com/shovel
MIT License
183 stars 23 forks source link

flag --print-schema not recognized #244

Closed vmaark closed 8 months ago

vmaark commented 8 months ago

According to the docs the following should print out the db schema: ./shovel -config minimal.json --print-schema

I'm getting the following error on version v1.0beta d80f:

 ../shovel/shovel -config shovel/config.json --print-schema
flag provided but not defined: -print-schema
Usage of ../shovel/shovel:
  -config string
        task config file
  -l string
        dashboard server listen address (default ":8546")
  -notx
        disable pg tx
  -pqxtest.d postgres -d
        postgres debug level (see postgres -d)
  -profile string
        run profile after indexing
  -skip-migrate
        do not run db migrations on startup
  -version
        version

Based on the output it looks like there's something wrong with parsing the args, as there's a - missing in the output.

ryandotsmith commented 8 months ago

I think that feature was added since 1.0beta. Can you please run the latest version on main? It is quite stable and is a candidate for v1.0 which will be released soon.

https://indexsupply.com/shovel/docs/#versions

vmaark commented 8 months ago

Oh right, I installed the beta because the example in the docs uses that link. I downloaded now from the main, but it just exits immediately?

(base) ➜  shovel ./shovel --version
[1]    9300 killed     ./shovel

platform is correct (m1 macbook pro):

(base) ➜  shovel file shovel
shovel: Mach-O 64-bit executable arm64
ryandotsmith commented 8 months ago

These are the exact steps that I ran on my machine (mac arm64)

% uname -a                                                        
Darwin m4.local 23.4.0 Darwin Kernel Version 23.4.0: Wed Feb 21 21:44:43 PST 2024; root:xnu-10063.101.15~2/RELEASE_ARM64_T6000 arm64
% curl -LO --silent https://indexsupply.net/bin/main/darwin/arm64/shovel

% chmod +x shovel                                                          
% ./shovel --version                                                       
vmain 0ec8

Do you get a different result?

vmaark commented 8 months ago 
(base) ➜  shovel uname -a
Darwin vmark.local 21.3.0 Darwin Kernel Version 21.3.0: Wed Jan  5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_ARM64_T6000 arm64
(base) ➜  shovel curl -LO --silent https://indexsupply.net/bin/main/darwin/arm64/shovel
(base) ➜  shovel chmod +x shovel
(base) ➜  shovel ./shovel --version
[1]    16331 killed     ./shovel --version
ryandotsmith commented 8 months ago

Can we verify that you are getting the correct binary?

% cat shovel | md5 
5a91e0f1f6f94da4d48c2052e56022a5

What shell are you running?

% echo "$SHELL"
/bin/zsh

What's the exit status?

% ./shovel --version
vmain 0ec8
% echo $?
0

This is an odd one. I've had many people download and install shovel on both mac and linux and this is the first that i'm seeing this.

ryandotsmith commented 8 months ago

Does building from source work?

  1. https://go.dev/doc/install
% go run github.com/indexsupply/x/cmd/shovel@main -version
go: downloading github.com/indexsupply/x v0.1.9-0.20240307171939-0ec8ce411db9
v 
% echo $?                                                 
0
vmaark commented 8 months ago

md5 is same as yours. Exit status 137

(base) ➜  shovel cat shovel | md5
5a91e0f1f6f94da4d48c2052e56022a5
(base) ➜  shovel echo $SHELL
/bin/zsh
(base) ➜  shovel  ./shovel --version
[1]    18771 killed     ./shovel --version
(base) ➜  shovel echo $?
137

btw the 1.0beta binary still works.

ryandotsmith commented 8 months ago

Exit status 137 seems to indicate that the OS is killing the process.

ryandotsmith commented 8 months ago

Can you confirm that you can run using the Go toolchain?

https://github.com/indexsupply/code/issues/244#issuecomment-1987041011

vmaark commented 8 months ago

go run github.com/indexsupply/x/cmd/shovel@main -version

No I can't. 

(base) ➜  shovel go run github.com/indexsupply/x/cmd/shovel@main -version
go: downloading github.com/indexsupply/x v0.1.9-0.20240307171939-0ec8ce411db9
go: downloading blake.io/pqx v0.2.1
go: downloading github.com/jackc/pgx/v5 v5.3.1
go: downloading filippo.io/age v1.0.0
go: downloading github.com/kr/session v0.2.1
go: downloading github.com/holiman/uint256 v1.2.3
go: downloading github.com/btcsuite/btcd/btcec/v2 v2.3.2
go: downloading golang.org/x/crypto v0.18.0
go: downloading github.com/golang/snappy v0.0.4
go: downloading github.com/goccy/go-json v0.10.2
go: downloading github.com/klauspost/compress v1.17.4
go: downloading nhooyr.io/websocket v1.8.10
go: downloading github.com/jackc/puddle/v2 v2.2.0
go: downloading github.com/jackc/pgpassfile v1.0.0
go: downloading github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a
go: downloading golang.org/x/text v0.14.0
go: downloading github.com/decred/dcrd/dcrec/secp256k1/v4 v4.1.0
go: downloading github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8
go: downloading kr.dev/errorfmt v0.1.1
../../../go/pkg/mod/github.com/indexsupply/x@v0.1.9-0.20240307171939-0ec8ce411db9/shovel/config/config.go:4:2: package cmp is not in GOROOT (/opt/homebrew/Cellar/go/1.20.6/libexec/src/cmp)
note: imported by a module that requires go 1.21
../../../go/pkg/mod/github.com/indexsupply/x@v0.1.9-0.20240307171939-0ec8ce411db9/dig/dig.go:12:2: package log/slog is not in GOROOT (/opt/homebrew/Cellar/go/1.20.6/libexec/src/log/slog)
note: imported by a module that requires go 1.21
../../../go/pkg/mod/github.com/indexsupply/x@v0.1.9-0.20240307171939-0ec8ce411db9/dig/dig.go:13:2: package slices is not in GOROOT (/opt/homebrew/Cellar/go/1.20.6/libexec/src/slices)
note: imported by a module that requires go 1.21

Maybe there's a dependency which was introduced between Darwin Kernel Version 21.3.0 (my version) and Darwin Kernel Version 23.4.0 (yours)?

Actually it looks like the go version?

ryandotsmith commented 8 months ago

Did you install the latest version of Go? The link I referenced is at 1.22.

Can you run this command as well:

% sudo dtruss -a ./shovel --version
dtrace: system integrity protection is on, some features will not be available

    PID/THRD  RELATIVE  ELAPSD    CPU SYSCALL(args)          = return
vmain 0ec8

CALL                                        COUNT
vmaark commented 8 months ago

Here's the output:

(base) ➜  shovel sudo dtruss -a ./shovel --version
dtrace: system integrity protection is on, some features will not be available

    PID/THRD  RELATIVE  ELAPSD    CPU SYSCALL(args)          = return

CALL                                        COUNT

I installed now the latest go, and I get the same output as you:

(base) ➜  shovel go run github.com/indexsupply/x/cmd/shovel@main -version
v
(base) ➜  shovel echo $?
0
(base) ➜  shovel
ryandotsmith commented 8 months ago

I just compiled a program named foobar that is 

package main

import "fmt"

func main() {
    fmt.Println("foobar")
}

I also compiled shovel on my laptop (instead of the build servers that update https://indexsupply.net/bin).

Can you unzip and run both of these programs and report the output?

Archive.zip

% cat Archive.zip| md5 
cd4f7b18cacd50094817c97f3fdfe7ba

% unzip Archive.zip     
Archive:  Archive.zip
  inflating: foobar                  
  inflating: shovel                  
  inflating: __MACOSX/._shovel

% ./foobar 
foobar

% ./shovel -version
v 0ec8
vmaark commented 8 months ago

I just compiled a program named foobar that is

package main

import "fmt"

func main() {
    fmt.Println("foobar")
}

I also compiled shovel on my laptop (instead of the build servers that update https://indexsupply.net/bin).

Can you unzip and run both of these programs and report the output?

Archive.zip

% cat Archive.zip| md5 
cd4f7b18cacd50094817c97f3fdfe7ba

% unzip Archive.zip     
Archive:  Archive.zip
  inflating: foobar                  
  inflating: shovel                  
  inflating: __MACOSX/._shovel

% ./foobar 
foobar

% ./shovel -version
v 0ec8

had to allow both in Security & Privacy, but otherwise got the same output:

(base) ➜  Archive ./foobar
foobar
(base) ➜  Archive ./shovel -version
v 0ec8
(base) ➜  Archive cat ../Archive.zip | md5
cd4f7b18cacd50094817c97f3fdfe7ba
vmaark commented 8 months ago

@ryandotsmith found this issue googling: https://github.com/golang/go/issues/63997

vmaark commented 8 months ago

@ryandotsmith found this issue googling: golang/go#63997

suspicious, found some logs:

default 20:25:41.259177+0100    kernel  ignoring detached code signature on 'shovel' with cdhash '6a486408671c111b08c20f2c60f5408b915a179d' because it is invalid, or not a simple adhoc signature.
default 20:25:41.259332+0100    kernel  CODE SIGNING: process 39392[shovel]: rejecting invalid page at address 0x104d58000 from offset 0x0 in file "/Users/vmark/workspace/trash.pub/shovel/shovel" (cs_mtime:1709898376.232402220 != mtime:1710063195.692618497) (signed:1 validated:1 tainted:1 nx:0 wpmapped:0 dirty:0 depth:0)
default 20:25:41.259392+0100    kernel  shovel[39392] Corpse allowed 1 of 5
default 20:25:45.488975+0100    ReportCrash Formulating fatal 309 report for corpse[39392] shovel
default 20:25:45.503654+0100    osanalyticshelper   creating type 309 as /Users/vmark/Library/Logs/DiagnosticReports/.shovel-2024-03-10-202545.ips
default 20:25:45.509876+0100    osanalyticshelper   Saved type '309(<private>)' report (16 of max 25) at /Users/vmark/Library/Logs/DiagnosticReports/shovel-2024-03-10-202545.ips
vmaark commented 8 months ago

I updated my MacOS from Monterey to Sonoma, it solves the issue, but probably worth looking into the codesigning

ryandotsmith commented 8 months ago

OK. Thanks for helping debug. I will keep an eye out on this.