wrgl pull

[ayyubibrahimi]

hey @pckhoi, I'm still receiving an error when I run wrgl pull --all. Please see below

ayyubi@DESKTOP-BLLRMEO:/mnt/c/Users/PC/Desktop/launch/processing$ wrgl pull --all
Pulling branches 0 / 195 [                                                                                                           ] 0 %  39m26s
error fetching objects: error receiving objects: pr.ReadObject error: stream error: stream ID 11; PROTOCOL_ERROR; received from peer
ayyubi@DESKTOP-BLLRMEO:/mnt/c/Users/PC/Desktop/launch/processing$ wrgl pull --all
Pulling branches 0 / 195 [                                                                                                             ] 0 %  7m0s
error fetching objects: error receiving objects: pr.ReadObject error: stream error: stream ID 11; PROTOCOL_ERROR; received from peer

[pckhoi]

Again I suspect this is due to unstable network? Try again when you have stable connection. Also, you can add the flag "--ignore-non-existent" if you ever add a new branch for the command to ignore that new branch.

[ayyubibrahimi]

Hmm. I'm back in NY with decent wifi. I'll continue to explore on my end.

[pckhoi]

Okay, let me know if you still encounter it. I certainly did run successfully on my local computer.

[ayyubibrahimi]

@pckhoi No luck. I assume that it's my computer that is the problem. This is problematic because I'm not able to exhaust every option to reproduce the error in #410. I'm back in New Orleans soon where I'll work on getting a new computer.

[pckhoi]

So I don't really expect or anticipate any usage of wrgl locally to be necessary to contribute productively. At least that's how I designed it. If the errors are strictly in the wrgl pull and commit stage then feel free to tell me to look at it directly. I figure expecting you to handle that part is just unreasonable.

Re getting a new computer, 100%, please get a machine with at least 16Gb of RAM.

[ayyubibrahimi]

Sounds good. Thanks.

[ayyubibrahimi]

Hey @pckhoi. Getting a new error at the wrgl pull stage of the process-data check. Please review PR #438 at your convenience. Thanks!

[pckhoi]

Yes, it is part of the migration to self-host wrgld. I will update this repo after the python sdk is updated.

[ayyubibrahimi]

Hey @pckhoi. Hope you had a good weekend. I'm receiving the error below which I believe is associated with an inability to pull the person table from wrgl. Are you receiving this error or could it be attributable to my computer? Thanks.

file build/person.csv is up to date.
running fuse/cross_agency.py build/person.csv
    Traceback (most recent call last):
      File "/mnt/c/Users/PC/Desktop/launch/processing/fuse/cross_agency.py", line 285, in <module>
        old_person_df = pd.read_csv(args.person_csv)
      File "/home/ayyubi/.local/lib/python3.9/site-packages/pandas/util/_decorators.py", line 211, in wrapper
        return func(*args, **kwargs)
      File "/home/ayyubi/.local/lib/python3.9/site-packages/pandas/util/_decorators.py", line 331, in wrapper
        return func(*args, **kwargs)
      File "/home/ayyubi/.local/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 950, in read_csv
        return _read(filepath_or_buffer, kwds)
      File "/home/ayyubi/.local/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 605, in _read
        parser = TextFileReader(filepath_or_buffer, **kwds)
      File "/home/ayyubi/.local/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1442, in __init__
        self._engine = self._make_engine(f, self.engine)
      File "/home/ayyubi/.local/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1753, in _make_engine
        return mapping[engine](f, **self.options)
      File "/home/ayyubi/.local/lib/python3.9/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 79, in __init__
        self._reader = parsers.TextReader(src, **kwds)
      File "pandas/_libs/parsers.pyx", line 554, in pandas._libs.parsers.TextReader.__cinit__
    pandas.errors.EmptyDataError: No columns to parse from file
make: *** [wrgl.mk:12: data/fuse/person.csv] Error 1

[pckhoi]

can you try make -B build/person.csv and show me the output?

[ayyubibrahimi]

ayyubi@DESKTOP-BLLRMEO:/mnt/c/Users/PC/Desktop/launch/processing$ make -B build/person.csv
.deba/deps/fuse.d:52: warning: overriding recipe for target 'data/fuse/post_carencro_pd.csv'
.deba/deps/fuse.d:46: warning: ignoring old recipe for target 'data/fuse/post_carencro_pd.csv'        
.deba/deps/fuse.d:52: warning: overriding group membership for target 'data/fuse/post_carencro_pd.csv'
make: [deba.mk:42: .deba] Error 1 (ignored)
make: [deba.mk:44: .deba/deps] Error 1 (ignored)
python -m deba deps --stage analysis
python -m deba deps --stage fuse
    no target, skipping script cross_agency.py
WARNING: prerequisite "clean/pprr_post_2020_11_06.csv" of script fuse/greenwood_pd.py found more than once
python -m deba deps --stage match
WARNING: prerequisite "clean/cprr_new_orleans_da_2016_2020.csv" of script match/new_orleans_pd.py found more than once
python -m deba deps --stage clean
    no prerequisite or reference, skipping script baton_rouge_fpcsb_lprr.py
python -m deba deps --stage ner
python -m deba deps --stage ocr
python -m deba deps --stage meta
python -m deba deps
.deba/deps/fuse.d:52: warning: overriding recipe for target 'data/fuse/post_carencro_pd.csv'
.deba/deps/fuse.d:46: warning: ignoring old recipe for target 'data/fuse/post_carencro_pd.csv'
.deba/deps/fuse.d:52: warning: overriding group membership for target 'data/fuse/post_carencro_pd.csv'
make: Nothing to be done for 'build/person.csv'.

[pckhoi]

Sorry, that wasn't the right command. The right command is make pull_person. But even that command stalled for me. I found out that wrgl pull was asking me to log in. So I did and now it seems alright.

$ wrgl pull person origin +refs/heads/person:refs/remotes/origin/person
Visit https://keycloak.llead.co/realms/llead-wrgl/device in your browser and enter user code "FBNG-OVQN" to login
Saved credentials to /Users/khoipham/.config/wrgl/credentials.yaml
Already up to date.
$ make pull_person                                                     
.deba/deps/fuse.d:52: warning: overriding recipe for target 'data/fuse/post_carencro_pd.csv'
.deba/deps/fuse.d:46: warning: ignoring old recipe for target 'data/fuse/post_carencro_pd.csv'
.deba/deps/fuse.d:52: warning: overriding group membership for target 'data/fuse/post_carencro_pd.csv'
file build/person.csv is up to date.

For the record, running wrgl pull person without specifying remote and refspec should be okay too.

Can you please run those again and show me the output. Particularly the output of wrgl pull? Also let me know which branch you are working off of.

[ayyubibrahimi]

My ubuntu crashes because I run out of space when I run this command. It's my computer. Please disregard until I can get a new computer. Hopefully sometime next week.