Build gets into strange state when first invocation uses wrong version of Python

[yyh-graphsql]

[@localhost grumpy]$ make build/src/grumpy/lib/itertools/module.go:5: can't find import: "grumpy/lib/sys" make: *** [build/pkg/linux_amd64/grumpy/lib/itertools.a] Error 1

[trotterdylan]

Weird. Can you paste the output of:

$ ls -la build/src/grumpy/lib/sys

[yyh-graphsql]

[z@localhost gium]$ cd ~/work/grumpy/ [z@localhost grumpy]$ ls -la build/src/grumpy/lib/sys total 4 drwxrwxr-x. 2 z z 21 Jan 4 19:03 . drwxrwxr-x. 41 z z 4096 Jan 4 19:03 .. -rw-rw-r--. 1 z z 0 Jan 4 19:03 module.d

[yyh-graphsql]

anything wrong @trotterdylan

[trotterdylan]

Okay so the .go file didn't get generated. It may be because the dependencies are incorrect. Here's what I see in the deps file for itertools:

$ cat build/src/grumpy/lib/itertools/module.d build/pkg/darwin_amd64/grumpy/lib/itertools.a: build/pkg/darwin_amd64/grumpy/lib/_collections.a build/pkg/darwin_amd64/grumpy/lib/itertools.a: build/pkg/darwin_amd64/grumpy/lib/sys.a

What do you see in build/src/grumpy/lib/itertools/module.d?

[yyh-graphsql]

It is an empty file @trotterdylan

[trotterdylan]

What's your system Python? I suspect things will not work unless it's 2.7. Mine is:

$ python --version Python 2.7.10

[yyh-graphsql]

[z@localhost z]$ python -V Python 2.7.5

[yyh-graphsql]

Any python module I have to install? @trotterdylan

[trotterdylan]

No, shouldn't need any Python modules besides what ships with the interpreter. It's possible there are differences in the modulefinder library between 2.7.5 and 2.7.10.

I'll see if I can reproduce the issue with 2.7.5, but I probably won't have a chance to get to it for a couple days. In the mean time, you could try installing 2.7.10+ to see if that helps.

[llazzaro]

Same issue here.

$ cat build/src/grumpy/lib/itertools/module.d build/pkg/_/grumpy/lib/itertools.a: build/pkg/_/grumpy/lib/_collections.a build/pkg/_/grumpy/lib/itertools.a: build/pkg/_/grumpy/lib/sys.a

python --version

Python 2.7.13 :: Anaconda custom (x86_64)

Also tried with non anaconda python:

Python 2.7.11

Same error

[trotterdylan]

@llazzaro Hmm it looks like a different issue because build/src/grumpy/lib/itertools/module.d is empty for yyh-graphsql. Are you seeing:

build/src/grumpy/lib/itertools/module.go:5: can't find import: "grumpy/lib/sys"

It looks like your Go architecture is off. What's the result of:

$ go env

[llazzaro]

the make command gives the following error:

build/src/grumpy/lib/itertools/module.go:5: can't find import: "grumpy/lib/sys"
make: *** [build/pkg/darwin_amd64/grumpy/lib/itertools.a] Error 1

Got this result after execute go env

$  go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/leonardolazzaro/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/1f/pdmszzls4xs6px9gmf627ng40000gn/T/go-build055261285=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"

[trotterdylan]

@llazzaro Would you mind pasting the output of this command run from the root of the repo:

$ python -m modulefinder -p $PWD/lib:$PWD/third_party/stdlib lib/itertools.py

[llazzaro]

Sure. Here is the output:

 $ python -m modulefinder -p $PWD/lib:$PWD/third_party/stdlib lib/itertools.py

  Name                      File
  ----                      ----
m __main__                  lib/itertools.py
m _collections              /Users/leonardolazzaro/workspace/grumpy/lib/_collections.py
m sys

[trotterdylan]

Okay, that looks correct. Maybe the issue is on the awk part of things. Try:

$ python -m modulefinder -p $PWD/lib:$PWD/third_party/stdlib lib/itertools.py | awk '{if (($1 == "m" || $1 == "P") && $2 != "__main__" && $2 != "itertools") {gsub(/\./, "/", $2); print "build/pkg/darwin_amd64/grumpy/lib/itertools.a: build/pkg/darwin_amd64/grumpy/lib/" $2 ".a"}}'

[llazzaro]

Output:

$ python -m modulefinder -p $PWD/lib:$PWD/third_party/stdlib lib/itertools.py | awk '{if (($1 == "m" || $1 == "P") && $2 != "__main__" && $2 != "itertools") {gsub(/\./, "/", $2); print "build/pkg/darwin_amd64/grumpy/lib/itertools.a: build/pkg/darwin_amd64/grumpy/lib/" $2 ".a"}}'
build/pkg/darwin_amd64/grumpy/lib/itertools.a: build/pkg/darwin_amd64/grumpy/lib/_collections.a
build/pkg/darwin_amd64/grumpy/lib/itertools.a: build/pkg/darwin_amd64/grumpy/lib/sys.a

[trotterdylan]

Huh, that all looks right. That output should appear verbatim in build/src/grumpy/lib/itertools/module.d which specifies sys.a as a dependency of itertools.a so sys will be built first.

Something's broken in the dependency chain there. You could probably manually get around by build sys.a manually: make build/pkg/darwin_amd64/grumpy/lib/sys.a and then make build/pkg/darwin_amd64/grumpy/lib/itertools.a.

[trotterdylan]

Sorry I haven't got anything more satisfying for you at the moment. I'll take a look at this again tomorrow.

[TomasTomecek]

I bumped into the same issue. Had to manually build a dozen of modules, thanks to suggestion by @trotterdylan. A Dockerfile would be really handy, can provide one once you resolve the dependency issue.

[llazzaro]

@trotterdylan I did the manual make. Got this new error:

$ make
build/src/grumpy/lib/itertools_test/module.go:5: can't find import: "grumpy/lib/weetest"
make: *** [build/pkg/darwin_amd64/grumpy/lib/itertools_test.a] Error 1

But this time I got some modules missing:

$ python -m modulefinder -p $PWD/lib:$PWD/third_party/stdlib  ./lib/weetest.py

  Name                      File
  ----                      ----
m __main__                  ./lib/weetest.py
m linecache                 /Users/leonardolazzaro/workspace/grumpy/third_party/stdlib/linecache.py
P os                        /Users/leonardolazzaro/workspace/grumpy/lib/os/__init__.py
m os.path                   /Users/leonardolazzaro/workspace/grumpy/lib/os/path.py
m stat                      /Users/leonardolazzaro/workspace/grumpy/lib/stat.py
m sys
m time                      /Users/leonardolazzaro/workspace/grumpy/lib/time.py
m traceback                 /Users/leonardolazzaro/workspace/grumpy/third_party/stdlib/traceback.py
m types                     /Users/leonardolazzaro/workspace/grumpy/third_party/stdlib/types.py

Missing modules:
? __go__.grumpy imported from os
? __go__.os imported from os, stat
? __go__.path.filepath imported from os, os.path
? __go__.syscall imported from os
? __go__.time imported from os, time

[S-YOU]

@TomasTomecek I have made a docker image on #37 in case you interested

[toxinu]

It happen to me when I first run make run while using python 3.x, then switch to python 2.x and re-run make run. I don't know which temporary files to remove, so I just clone again Grumpy.

[S-YOU]

You could run make clean, to clean up build folder.

[hiemal]

Any idea of this... ?

build/src/grumpy/lib/os/path_test/module.go:4: can't find import: "grumpy/lib/os

I suspect it's a PATH issue...

---

Ubuntu 14.04 with GO path set.

[mskims]

@hiemal you could config GOPATH link

[hiemal]

@mskims Thanks but this is not the issue. Go env is correctly setup on my machine... This is an issue of grumpy not Go

[trotterdylan]

As suggested by S-YOU, it's possible that a make clean will help here.

Anybody who's having this problem that's willing, please delete the build/ subdir with rm -rf build and then paste the output of the following:

python -V
go version
make -v

Finally dump the output of make -d. I recommend putting it in pastebin or somewhere because it will be very large.

[hiemal]

make clean works. ty

[trotterdylan]

Thanks for letting me know. Do you happen to remember what steps lead you to encounter the issue in the first place? Did you have to reinstall Python or something?

[selik]

@trotterdylan

$ rm -rf build

$ python -V
Python 2.7.12 :: Continuum Analytics, Inc.

$ go version
go version go1.7.4 darwin/amd64
$ make -v
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i386-apple-darwin11.3.0

$ make -d > make-d.log
line 723: binary op not implemented: Pow
make: *** [build/src/grumpy/lib/collections/module.go] Error 2

I'd share the output of make -d but pastebin told me "You have exceeded the maximum paste size of 512 kilobytes per paste."

I also got the itertools errors at first, then ran make clean and started to get different errors. I also probably ran things accidentally in a Py3k environment, then switched to Python 2.

[trotterdylan]

@selik that's a different error. I think you might be encountering it because you've uncommented a "x ** y" expression in your PR https://github.com/google/grumpy/pull/44/files#diff-e338d6618c0a8dd1dd11b513c8e414cbR727

[selik]

@trotterdylan You're correct. I commented the ** line and no more errors.

[llazzaro]

@trotterdylan my issue was that python 3 is my default interpreter. I did a make and then switched to python 2. This was solved doing a make clean.

[trotterdylan]

Great. Thanks for the update. I'm going to add some checks to the Makefile to verify the correct Python and Go versions.

[trotterdylan]

I'm going to close this in favor of https://github.com/google/grumpy/issues/21 which has a nicer explanation of the solution.

[drgrib]

@trotterdylan

I'm still getting this in the form of

build/src/grumpy/lib/itertools/module.go:4: can't find import: "grumpy/lib/_collections"
make: *** [build/pkg/darwin_amd64/grumpy/lib/itertools.a] Error 1

This thread is very long with many possible routes, so I'm not sure how to proceed. The solution suggested by the OP of #21 seems obsolete since the makefile has changed. And #88 doesn't seem to have fixed this for me.

I have the following specifications:

$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.12.3
BuildVersion:   16D32

$ go version
go version go1.7.4 darwin/amd64

$ python2.7 -V
Python 2.7.10

$ python3 -V
Python 3.5.2

[trotterdylan]

Have you tried "make clean; make"? On Fri, Feb 3, 2017 at 11:28 PM Chris Redford notifications@github.com wrote:

Still getting this in the form:

build/src/grumpy/lib/itertools/module.go:4: can't find import: "grumpy/lib/_collections" make: *** [build/pkg/darwin_amd64/grumpy/lib/itertools.a] Error 1

This thread is very long with many possible routes, so I'm not sure how to proceed. The solution suggested by the OP of #21 https://github.com/google/grumpy/issues/21 seems obsolete since the makefile has changed.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/google/grumpy/issues/24#issuecomment-277425776, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHLe2cbAhCu4_wl5kCPXnWOBEKiy5LZks5rZCiLgaJpZM4LbNeR .

[drgrib]

@trotterdylan Several times, yes. No fix, unfortunately.

make after a make clean causes a huge succession of repeated lines like this:

IOError: [Errno 2] No such file or directory: 'grumpyTest/grumpy/third_party/stdlib:/Users/credford/Dropbox/Years/__2017/__Scripts/Professional/8'
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/modulefinder.py", line 671, in <module>
    mf = test()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/modulefinder.py", line 663, in test
    mf.load_file(arg)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/modulefinder.py", line 119, in load_file
    with open(pathname, READ_MODE) as fp:
IOError: [Errno 2] No such file or directory: 'grumpyTest/grumpy/third_party/stdlib:/Users/credford/Dropbox/Years/__2017/__Scripts/Professional/8'
build/src/grumpy/lib/itertools/module.go:4: can't find import: "grumpy/lib/_collections"
make: *** [build/pkg/darwin_amd64/grumpy/lib/itertools.a] Error 1

I use python3 heavily in my system, if that helps.

[cclauss]

The CWD contains a ':' AND is in Dropbox. Could either of those be a problem? Did you try make install? Do you use homebrew?

[drgrib]

@cclauss

Good spotting on the path. There wasn't a : in the path. Rather, there was a space. The code being run by the makefile was mutating the path and appended the beginning to the end.

Specifically, the makefile was transforming the path from

/Users/credford/Dropbox/Years/__2017/__Scripts/Professional/8 grumpyTest/grumpy/third_party/stdlib

(notice the space after the 8) to

grumpyTest/grumpy/third_party/stdlib:/Users/credford/Dropbox/Years/__2017/__Scripts/Professional/8

Changing the name of the offending folder from 8 grumpyTest to 8-grumpyTest relieved the error. You may want to make your makefile more robust for spaces in paths, though.

I've been running code, including many Go and Python programs, in Dropbox for a year with no issues. I have several git repositories within it and make several commits a day.

It looks like the make has been successful, so thank you.

However, I would much rather manage this through homebrew, which I do use, yes. Is there a recipe? If so, could you tell me what it is?

[cclauss]

The homebrew formula is here but it is not yet accepted by the homebrew crew until grumpy has a release. However the formula works if you put it in place by hand and then do brew install grumpy --HEAD.

[trotterdylan]

I was sort of aware that spaces in filenames would be problematic. Thanks for diagnosing this. @drgrib would you mind filing another issue specifically about that with some of the information gathered here?

[drgrib]

@trotterdylan Sure thing.