search

indygreg / PyOxidizer

A modern Python application packaging and distribution tool
Mozilla Public License 2.0
5.48k stars 239 forks source link

AttributeError: 'NoneType' object has no attribute 'loader' #100

Open monocongo opened 5 years ago

monocongo commented 5 years ago

I have built a PyOxidizer project and when I run the executable file I get the following error:

AttributeError: 'NoneType' object has no attribute 'loader'
error: cargo run failed

Can someone suggest what I can do to get more information, perhaps an option for pyoxidizer run that provides more verbose output?

My pyoxidizer.toml file:

[[build]]
application_name = "deep_monitor"

[[embedded_python_config]]
raw_allocator = "jemalloc"

[[embedded_python_config]]
build_target = "x86_64-pc-windows-msvc"
raw_allocator = "system"

[[packaging_rule]]
type = "stdlib-extensions-policy"
policy = "all"

[[packaging_rule]]
type = "stdlib"
include_source = false

[[packaging_rule]]
type = "write-license-files"
path = ""

[[packaging_rule]]
type = "virtualenv"
path = "/home/james/.virtualenvs/pyox"

[[embedded_python_run]]
mode = "module"
module = "deep_monitor"

[[python_distribution]]
build_target = "x86_64-apple-darwin"
url = "https://github.com/indygreg/python-build-standalone/releases/download/20190617/cpython-3.7.3-macos-20190618T0523.tar.zst"
sha256 = "6668202a3225892ce252eff4bb53a58ac058b6a413ab9d37c026a500c2a561ee"
[[python_distribution]]
build_target = "x86_64-pc-windows-msvc"
url = "https://github.com/indygreg/python-build-standalone/releases/download/20190617/cpython-3.7.3-windows-amd64-20190618T0516.tar.zst"
sha256 = "fd43554b5654a914846cf1c251d1ad366f46c7c4d20b7c44572251b533351221"
[[python_distribution]]
build_target = "x86_64-unknown-linux-gnu"
url = "https://github.com/indygreg/python-build-standalone/releases/download/20190617/cpython-3.7.3-linux64-20190618T0324.tar.zst"
sha256 = "d6b80a9723c124d6d193f8816fdb874ba6d56abfb35cbfcc2b27de53176d0620"
[[python_distribution]]
build_target = "x86_64-unknown-linux-musl"
url = "https://github.com/indygreg/python-build-standalone/releases/download/20190617/cpython-3.7.3-linux64-musl-20190618T0400.tar.zst"
sha256 = "2be2d109b82634b36685b89800887501b619ef946dda182e5a8ab5c7029a8136"

[[pyoxidizer]]
version = "0.2.0"
commit = ""

Thanks in advance for any comments or suggestions.

lvscar commented 5 years ago

I have encountered the similar problem.

In my case, my PyOxidizer generated executing file could ran smoothly in the environment that build it (Mac OS 10.14.5). But when I execute it in old version of MacOS (both 10.14.3 & 10.13.6) I got the same Error . ''' AttributeError: 'NoneType' object has no attribute 'loader' '''

indygreg commented 5 years ago

Was there any other context to the error message? Do you know what module import was raising this error?

lvscar commented 5 years ago

When the error message was raising on target machine, there were no more error messages that revealing valuable debug clue.

In my case, there are some packaging rule I specified as install_location = "app-relative:lib". I will test if is the problem caused by install location specifying.

indygreg commented 5 years ago

Is there any chance I could get the source code to deep_monitor (or at least its list of import statements) so I can try to reproduce this?

monocongo commented 5 years ago

@indygreg thanks for looking into this for us, and for your work on this project.

I apologize for being unable to provide the source for my deep_monitor project at this time. The requirements have likely changed since I submitted this issue and I should perhaps also try to reproduce this issue with the current requirements.txt. Below are the various import statements from the source files, as well as the requirements.

import argparse
import configparser
import csv
import logging
import os
import pickle
import shutil
import sqlalchemy
from sqlalchemy import (
    Column,
    DateTime,
    Float,
    ForeignKey,
    Integer,
    LargeBinary,
    Unicode,
)
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import relationship
from sqlalchemy.sql import func
import sqlalchemy_utils
from sqlalchemy_utils import database_exists, create_database
import tempfile
import time
from typing import Dict, List

import arrow
import cv2
import imutils
from imutils.video import FPS
from imutils.video import VideoStream
from keras_retinanet.models import load_model
from keras_retinanet.utils.image import preprocess_image
from keras_retinanet.utils.image import resize_image
import numpy as np
import tensorflow as tf
import wget

requirements.txt:

absl-py==0.7.1
arrow==0.14.2
astor==0.8.0
Cython==0.29.12
gast==0.2.2
google-pasta==0.1.7
grpcio==1.22.0
h5py==2.9.0
imutils==0.5.2
joblib==0.13.2
Keras==2.2.4
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
keras-resnet==0.2.0
keras-retinanet==0.5.1
Markdown==3.1.1
numpy==1.16.4
opencv-python==4.1.0.25
Pillow==6.1.0
progressbar2==3.42.0
protobuf==3.9.0
python-dateutil==2.8.0
python-utils==2.3.0
PyYAML==5.1.1
scikit-learn==0.21.2
scipy==1.3.0
six==1.12.0
SQLAlchemy==1.3.6
SQLAlchemy-Utils==0.34.1
tensorboard==1.14.0
tensorflow==1.14.0
tensorflow-estimator==1.14.0
tensorflow-gpu==1.14.0
termcolor==1.1.0
Werkzeug==0.15.5
wget==3.2
wrapt==1.11.2
n8henrie commented 5 years ago

I am currently getting the AttributeError: 'NoneType' object has no attribute 'loader' error when trying to implement the filter-include rule to trim down the binary for https://github.com/n8henrie/nmfishingreport .

If I comment out the filter-include in pyoxidizer.toml things work fine. I tried to do a "module collection" run for every flag in my CLI app (in case different modules were imported).

Here's the concat / deduped content of my modules files:

__future__
__main__
_abc
_bisect
_blake2
_bootlocale
_bz2
_codecs
_collections
_collections_abc
_compression
_datetime
_frozen_importlib
_frozen_importlib_external
_functools
_hashlib
_heapq
_imp
_io
_locale
_lzma
_markupbase
_operator
_pyoxidizer_importer
_random
_scproxy
_sha3
_signal
_socket
_sqlite3
_sre
_ssl
_stat
_string
_strptime
_struct
_thread
_warnings
_weakref
_weakrefset
abc
argparse
atexit
base64
binascii
bisect
bs4
bs4.builder
bs4.builder._htmlparser
bs4.dammit
bs4.element
bs4.formatter
builtins
bz2
calendar
chardet
chardet.big5freq
chardet.big5prober
chardet.chardistribution
chardet.charsetgroupprober
chardet.charsetprober
chardet.codingstatemachine
chardet.compat
chardet.cp949prober
chardet.enums
chardet.escprober
chardet.escsm
chardet.eucjpprober
chardet.euckrfreq
chardet.euckrprober
chardet.euctwfreq
chardet.euctwprober
chardet.gb2312freq
chardet.gb2312prober
chardet.hebrewprober
chardet.jisfreq
chardet.jpcntx
chardet.langbulgarianmodel
chardet.langcyrillicmodel
chardet.langgreekmodel
chardet.langhebrewmodel
chardet.langthaimodel
chardet.langturkishmodel
chardet.latin1prober
chardet.mbcharsetprober
chardet.mbcsgroupprober
chardet.mbcssm
chardet.sbcharsetprober
chardet.sbcsgroupprober
chardet.sjisprober
chardet.universaldetector
chardet.utf8prober
chardet.version
codecs
collections
collections.abc
configparser
contextlib
copyreg
datetime
email
email._encoded_words
email._parseaddr
email._policybase
email.base64mime
email.charset
email.encoders
email.errors
email.feedparser
email.header
email.iterators
email.message
email.parser
email.quoprimime
email.utils
encodings
encodings.aliases
encodings.cp1252
encodings.idna
encodings.latin_1
encodings.utf_8
enum
errno
fnmatch
functools
genericpath
gettext
grp
hashlib
heapq
html
html.entities
html.parser
http
http.client
importlib
importlib._bootstrap
importlib._bootstrap_external
importlib.abc
importlib.machinery
importlib.util
io
itertools
keyword
linecache
locale
logging
lzma
marshal
math
nmfishingreport
nmfishingreport.internet_on
nmfishingreport.nmfishingreport
operator
os
os.path
posix
posixpath
pwd
quopri
random
re
reprlib
select
selectors
shutil
socket
soupsieve
soupsieve.__meta__
soupsieve.css_match
soupsieve.css_parser
soupsieve.css_types
soupsieve.util
sqlite3
sqlite3.dbapi2
sre_compile
sre_constants
sre_parse
ssl
stat
string
stringprep
struct
sys
tempfile
threading
time
token
tokenize
traceback
types
unicodedata
urllib
urllib.error
urllib.parse
urllib.request
urllib.response
uu
warnings
weakref
zipimport
zlib

EDIT: I also don't see any references to a .loader in my code, but there are a few in pip and pkg_resources:

$ rg -l -uuu -F '.loader' src/
$ rg -l -uuu -F '.loader' .venv/
.venv/lib/python3.7/site-packages/pkg_resources/__init__.py
.venv/lib/python3.7/site-packages/pip/_vendor/distlib/resources.py
.venv/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py
42sol-eu commented 5 years ago

I used the latest version of pyoxidizer and a venv (numpy is definitively installed). When I switch to a repl build i get 

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "numpy", line 126, in <module>
  File "numpy.__config__", line 9, in <module>
NameError: name '__file__' is not defined

does this help? how could we condense out the error codes that you need from the build process (any thin we should look for?)

RohitJones commented 4 years ago

@n8henrie I can confirm that I too am facing the same issue.

I encounter the error "AttributeError: 'NoneType' object has no attribute 'loader'" only when I have the FilterInclude packaging rule.

I've linked a small dummy project to demonstrate this issue. https://github.com/RohitJones/pyoxidizer_filter_include_issue

rafaelichow commented 4 years ago

Same issue here

wkschwartz commented 4 years ago

Update: I think the following is also relevant to #246

Reproducing the error

I have posted a minimal failing example at wkschwartz/pyoxidizer-attr-error to reproduce AttributeError: 'NoneType' object has no attribute 'loader'. Just clone it and execute pyoxidizer run. The resulting output should look like pyoxidizer-run-error.log (at the top of which I have added my machine info; I also replaced some directory names for readability).

You can prevent the AttributeError—but get no output or error message at all!—if you un-comment https://github.com/wkschwartz/pyoxidizer-attr-error/blob/fe5c3527f051d81186f794e0715b3c76c508b2c2/pyoxidizer.bzl#L17-L20

You can get the much less surprising error below if instead you leave the above lines commented out but swap which of these two lines are commented out: https://github.com/wkschwartz/pyoxidizer-attr-error/blob/fe5c3527f051d81186f794e0715b3c76c508b2c2/pyoxidizer.bzl#L6-L7

Traceback (most recent call last):
  File "importlib.util", line 94, in find_spec
ModuleNotFoundError: No module named 'pyox'
error: cargo run failed

If you make both changes, everything works as expected (almost: my original intent in making this example was to see what exceptions and tracebacks look like from PyOxidizer; I got the ValueError exception I was looking for, but also got a SystemError, which should never happen).

Analysis

This seems to be a problem of early error detection and error messaging. PyOxidizer's documentation does say that run_module works differently from python -m, but this is still a little surprising even when I knew to expect it. I think the reason I got no output when the package was included but run_module was set to pyox is that PyOxidizer runs pyox/__init__.py as __main__, and pyox/__init__.py is empty.

The problem that this issue is about though is when the package isn't included, which either generates the AttributeError or the ModuleNotFoundError. The latter is much less surprising and much more informative. However, since PyOxidizer has full control over which modules are available, I would think it would be possible for it to detect that the module listed in run_module isn't included at build time (and hopefully before the long compilation of jemalloc...)