Open jedie opened 9 years ago
Since https://github.com/jedie/pypyjs/commit/a5408122dd3418cb7f15c8b672a7252eb96400df i get better error messages on import errors...
Now i do some tests with the editor (from https://github.com/rfk/pypyjs/pull/102 ):
The selenium unittests for editor failed some tests...
e.g.: Test code:
import random
print random.__file__
will fail with:
*** Console output is: ***
Traceback (most recent call last):
File "c callback", line 3, in <module>
File "<string>", line 1, in <module>
ImportError: No module named random
But if i use the editor.html and write this code by hand, i get the right result:
/lib/pypyjs/lib_pypy/random.py
But sometimes, this failed and sometimes not... Strange!
A updated complete list of the import test is here: https://gist.github.com/jedie/b44b81260d3fcb90436f
You see also, that 'random' is fine here: https://gist.github.com/jedie/b44b81260d3fcb90436f#file-imports-py-L577-L578
With https://github.com/jedie/pypyjs/commit/84280878a88af70238d814cf06945f333d6b3c6d i check if all collected module_names exist in vm._allModules (btw. this is a list from pypy.js-0.3.0/lib/modules/index.json isn't it?)
And this worked. So IMHO all modules should be imported, excepted import problems like with CGIHTTPServer because of: ImportError: No module named _socket
Ah!
I think the importStatementRE in pypy.js is the problem: https://github.com/rfk/pypyjs/commit/0600a0d49a42c65183403bf04adae91a18390c0c#commitcomment-11279757
import random
print random.__file__
In this case: match[0] will be: "import random\nprint random.file" The imports value in return Promise.resolve(imports); is: [["import random\nprint random.file"]]
Hackish code to use ast:
import ast
tree = ast.parse("""from __future__ import absolute_import, division, print_function
import random
from foo import bar
import os.path as os_path
from mod import src as dst
print(random.__file__)""")
for node in ast.walk(tree):
if isinstance(node, ast.Import):
for name in node.names:
print("\nimport:", name.name)
if name.asname is not None:
print("\tas name:", name.asname)
elif isinstance(node, ast.ImportFrom):
print("\nfrom module:", node.module, "level:", node.level)
for name in node.names:
print("\tImport:", name.name)
if name.asname is not None:
print("\t\tas name:", name.asname)
results in:
from module: __future__ level: 0
Import: absolute_import
Import: division
Import: print_function
import: random
from module: foo level: 0
Import: bar
import: os.path
as name: os_path
from module: mod level: 0
Import: src
as name: dst
But, i have no idea to replace the JS stuff with the python ast stuff ;)
In this case: match[0] will be: "import random\nprint random.file" The imports value in return Promise.resolve(imports); is: [["import random\nprint random.file"]]
I have seen these output with:
--------------------- website/js/pypy.js-0.3.0/lib/pypy.js ---------------------
index d31bc2a..92fd4ca 100644
@@ -686,10 +686,14 @@ PyPyJS.prototype._repl_loop = function _repl_loop(prmpt, ps1) {
//
var importStatementRE = /(from\s+([a-zA-Z0-9_\.]+)\s+)?import\s+\(?\s*([a-zA-Z0-9_\.\*]+(\s+as\s+[a-zA-Z0-9_]+)?\s*,?\s*)+\s*\)?/g
PyPyJS.prototype.findImportedNames = function findImportedNames(code) {
var match = null;
var imports = [];
importStatementRE.lastIndex = 0;
while ((match = importStatementRE.exec(code)) !== null) {
+ window.vm.stderr("Import code match[0]: ["+JSON.stringify(match[0])+"]\n");
var relmod = match[2];
if (relmod) {
relmod = relmod + ".";
@@ -697,6 +701,7 @@ PyPyJS.prototype.findImportedNames = function findImportedNames(code) {
relmod = "";
}
var submods = match[0].split("import")[1];
+ window.vm.stderr("submods 1: ["+JSON.stringify(submods)+"]\n");
while (submods && /[\s(]/.test(submods.charAt(0))) {
submods = submods.substr(1);
}
@@ -707,9 +712,12 @@ PyPyJS.prototype.findImportedNames = function findImportedNames(code) {
for (var i = 0; i < submods.length; i++) {
var submod = submods[i];
submod = submod.split(/\s*as\s*/)[0];
+ window.vm.stderr("submods 2: ["+JSON.stringify(submods)+"]\n");
+ window.vm.stderr("relmod + submod: ["+JSON.stringify(relmod + submod)+"]\n");
imports.push(relmod + submod);
}
}
+ window.vm.stderr("imports: ["+JSON.stringify(imports)+"]\n");
return Promise.resolve(imports);
}
Here is the ast usage code, that does the "same" as PyPyJS.prototype.findImportedNames:
import ast
tree = ast.parse("""import sys
from os import path
import traceback as tb
from datetime import date as d""")
modules = []
for node in ast.walk(tree):
if isinstance(node, ast.Import):
modules += [name.name for name in node.names]
elif isinstance(node, ast.ImportFrom):
modules += [
"%s.%s" % (node.module, name.name)
for name in node.names
]
print(modules) # ['sys', 'os.path', 'traceback', 'datetime.date']
With https://github.com/jedie/pypyjs/commit/51274917c1956f295f214f0d060180a08d580288 i add a work-a-round for https://github.com/rfk/pypyjs/issues/127
Now two tests runs:
test_imports1
-----------------------------------
Console output:
-----------------------------------
1183 modules (unfiltered)
203 modules (filtered)
0 - import BaseHTTPServer ERROR: No module named BaseHTTPServer
1 - import Bastion ERROR: No module named Bastion
2 - import CGIHTTPServer ERROR: No module named CGIHTTPServer
3 - import ConfigParser ERROR: No module named ConfigParser
4 - import Cookie ERROR: No module named Cookie
5 - import DocXMLRPCServer ERROR: No module named DocXMLRPCServer
6 - import HTMLParser ERROR: No module named HTMLParser
7 - import MimeWriter ERROR: No module named MimeWriter
8 - import Queue ERROR: No module named Queue
9 - import SimpleHTTPServer ERROR: No module named SimpleHTTPServer
10 - import SimpleXMLRPCServer ERROR: No module named SimpleXMLRPCServer
11 - import SocketServer ERROR: No module named SocketServer
12 - import StringIO ERROR: No module named StringIO
13 - import UserDict OK
14 - import UserList ERROR: No module named UserList
15 - import UserString ERROR: No module named UserString
16 - import abc OK
17 - import aifc ERROR: No module named aifc
18 - import antigravity ERROR: No module named antigravity
19 - import anydbm ERROR: No module named anydbm
20 - import argparse ERROR: No module named argparse
21 - import ast ERROR: No module named ast
22 - import asynchat ERROR: No module named asynchat
23 - import asyncore ERROR: No module named asyncore
24 - import atexit ERROR: No module named atexit
25 - import base64 OK
26 - import bdb ERROR: No module named bdb
27 - import binhex ERROR: No module named binhex
28 - import bisect ERROR: No module named bisect
29 - import cPickle ERROR: No module named cPickle
30 - import cProfile ERROR: No module named cProfile
31 - import cStringIO OK
32 - import calendar ERROR: No module named calendar
33 - import cgi ERROR: No module named cgi
34 - import cgitb ERROR: No module named cgitb
35 - import chunk ERROR: No module named chunk
36 - import cmd ERROR: No module named cmd
37 - import code OK
38 - import codecs OK
39 - import codeop OK
40 - import collections ERROR: No module named collections
41 - import colorsys ERROR: No module named colorsys
42 - import commands ERROR: No module named commands
43 - import compileall ERROR: No module named compileall
44 - import compiler ERROR: No module named compiler
45 - import contextlib ERROR: No module named contextlib
46 - import cookielib ERROR: No module named cookielib
47 - import copy OK
48 - import csv ERROR: No module named csv
49 - import datetime ERROR: No module named datetime
50 - import dbhash ERROR: No module named dbhash
51 - import dbm ERROR: No module named dbm
52 - import decimal ERROR: No module named decimal
53 - import difflib ERROR: No module named difflib
54 - import dircache ERROR: No module named dircache
55 - import dis ERROR: No module named dis
56 - import distutils ERROR: No module named distutils
57 - import doctest ERROR: No module named doctest
58 - import dumbdbm ERROR: No module named dumbdbm
59 - import email ERROR: No module named email
60 - import encodings OK
61 - import filecmp ERROR: No module named filecmp
62 - import fileinput ERROR: No module named fileinput
63 - import fnmatch ERROR: No module named fnmatch
64 - import formatter ERROR: No module named formatter
65 - import fpformat ERROR: No module named fpformat
66 - import fractions ERROR: No module named fractions
67 - import ftplib ERROR: No module named ftplib
68 - import functools ERROR: No module named functools
69 - import gdbm ERROR: No module named gdbm
70 - import genericpath OK
71 - import getopt OK
72 - import getpass ERROR: No module named getpass
73 - import gettext ERROR: No module named gettext
74 - import glob ERROR: No module named glob
75 - import greenlet ERROR: No module named greenlet
76 - import grp ERROR: No module named grp
77 - import gzip ERROR: No module named gzip
78 - import hashlib ERROR: No module named hashlib
79 - import heapq ERROR: No module named heapq
80 - import hmac ERROR: No module named hmac
81 - import htmlentitydefs ERROR: No module named htmlentitydefs
82 - import htmllib ERROR: No module named htmllib
83 - import httplib ERROR: No module named httplib
84 - import ihooks ERROR: No module named ihooks
85 - import imaplib ERROR: No module named imaplib
86 - import imghdr ERROR: No module named imghdr
87 - import importlib ERROR: No module named importlib
88 - import imputil ERROR: No module named imputil
89 - import inspect ERROR: No module named inspect
90 - import io ERROR: No module named io
91 - import json ERROR: No module named json
92 - import keyword ERROR: No module named keyword
93 - import lib2to3 ERROR: No module named lib2to3
94 - import linecache OK
95 - import locale ERROR: No module named locale
96 - import logging ERROR: No module named logging
97 - import macurl2path ERROR: No module named macurl2path
98 - import mailbox ERROR: No module named mailbox
99 - import mailcap ERROR: No module named mailcap
100 - import markupbase ERROR: No module named markupbase
101 - import marshal OK
102 - import md5 ERROR: No module named md5
103 - import mhlib ERROR: No module named mhlib
104 - import mimetools ERROR: No module named mimetools
105 - import mimetypes ERROR: No module named mimetypes
106 - import mimify ERROR: No module named mimify
107 - import modulefinder ERROR: No module named modulefinder
108 - import multifile ERROR: No module named multifile
109 - import mutex ERROR: No module named mutex
110 - import netrc ERROR: No module named netrc
111 - import new ERROR: No module named new
112 - import nntplib ERROR: No module named nntplib
113 - import nturl2path ERROR: No module named nturl2path
114 - import numbers ERROR: No module named numbers
115 - import opcode ERROR: No module named opcode
116 - import optparse ERROR: No module named optparse
117 - import os OK
118 - import pdb ERROR: No module named pdb
119 - import pickle ERROR: No module named pickle
120 - import pickletools ERROR: No module named pickletools
121 - import pipes ERROR: No module named pipes
122 - import pkgutil ERROR: No module named pkgutil
123 - import platform ERROR: No module named platform
124 - import plistlib ERROR: No module named plistlib
125 - import popen2 ERROR: No module named popen2
126 - import poplib ERROR: No module named poplib
127 - import posixfile ERROR: No module named posixfile
128 - import posixpath OK
129 - import pprint ERROR: No module named pprint
130 - import profile ERROR: No module named profile
131 - import pstats ERROR: No module named pstats
132 - import pty ERROR: No module named pty
133 - import pwd ERROR: No module named ctypes_support
134 - import pyclbr ERROR: No module named pyclbr
135 - import pydoc ERROR: No module named pydoc
136 - import pyrepl ERROR: No module named pyrepl
137 - import quopri ERROR: No module named quopri
138 - import random ERROR: No module named random
139 - import re OK
140 - import repr OK
141 - import resource ERROR: No module named resource
142 - import rexec ERROR: No module named rexec
143 - import rfc822 ERROR: No module named rfc822
144 - import rlcompleter ERROR: No module named rlcompleter
145 - import robotparser ERROR: No module named robotparser
146 - import runpy ERROR: No module named runpy
147 - import sched ERROR: No module named sched
148 - import sets ERROR: No module named sets
149 - import sgmllib ERROR: No module named sgmllib
150 - import sha ERROR: No module named sha
151 - import shelve ERROR: No module named shelve
152 - import shlex ERROR: No module named shlex
153 - import shutil ERROR: No module named shutil
154 - import site ERROR: No module named site
155 - import smtpd ERROR: No module named smtpd
156 - import smtplib ERROR: No module named smtplib
157 - import sndhdr ERROR: No module named sndhdr
158 - import socket ERROR: No module named socket
159 - import sre ERROR: No module named sre
160 - import stackless ERROR: No module named stackless
161 - import stat OK
162 - import statvfs ERROR: No module named statvfs
163 - import string ERROR: No module named string
164 - import stringold ERROR: No module named stringold
165 - import stringprep ERROR: No module named stringprep
166 - import struct OK
167 - import symbol OK
168 - import sysconfig ERROR: No module named sysconfig
169 - import syslog ERROR: No module named syslog
170 - import tabnanny ERROR: No module named tabnanny
171 - import tarfile ERROR: No module named tarfile
172 - import telnetlib ERROR: No module named telnetlib
173 - import tempfile ERROR: No module named tempfile
174 - import test ERROR: No module named test
175 - import textwrap ERROR: No module named textwrap
176 - import this ERROR: No module named this
177 - import timeit ERROR: No module named timeit
178 - import toaiff ERROR: No module named toaiff
179 - import token OK
180 - import tokenize ERROR: No module named tokenize
181 - import tputil ERROR: No module named tputil
182 - import trace ERROR: No module named trace
183 - import traceback OK
184 - import tty ERROR: No module named tty
185 - import types OK
186 - import unittest ERROR: No module named unittest
187 - import urllib ERROR: No module named urllib
188 - import urllib2 ERROR: No module named urllib2
189 - import urlparse ERROR: No module named urlparse
190 - import user ERROR: No module named user
191 - import uu ERROR: No module named uu
192 - import uuid ERROR: No module named uuid
193 - import warnings OK
194 - import weakref OK
195 - import webbrowser ERROR: No module named webbrowser
196 - import whichdb ERROR: No module named whichdb
197 - import wsgiref ERROR: No module named wsgiref
198 - import xdrlib ERROR: No module named xdrlib
199 - import xml ERROR: No module named xml
200 - import xmllib ERROR: No module named xmllib
201 - import xmlrpclib ERROR: No module named xmlrpclib
202 - import zipfile ERROR: No module named zipfile
25 worked imports - 178 failed imports
test_imports2
-----------------------------------
Console output:
-----------------------------------
1183 modules (unfiltered)
203 modules (filtered)
0 - import BaseHTTPServer ERROR: No module named _socket
1 - import Bastion ERROR: No module named Bastion
2 - import CGIHTTPServer ERROR: No module named _socket
3 - import ConfigParser OK
4 - import Cookie OK
5 - import DocXMLRPCServer ERROR: No module named _socket
6 - import HTMLParser OK
7 - import MimeWriter OK
8 - import Queue ERROR: No module named threading
9 - import SimpleHTTPServer ERROR: No module named _socket
10 - import SimpleXMLRPCServer ERROR: No module named _socket
11 - import SocketServer ERROR: No module named _socket
12 - import StringIO OK
13 - import UserDict OK
14 - import UserList OK
15 - import UserString OK
16 - import abc OK
17 - import aifc OK
18 - import antigravity ERROR: No module named subprocess
19 - import anydbm OK
20 - import argparse OK
21 - import ast ERROR: No module named ast
22 - import asynchat ERROR: No module named _socket
23 - import asyncore ERROR: No module named select
24 - import atexit OK
25 - import base64 OK
26 - import bdb OK
27 - import binhex OK
28 - import bisect OK
29 - import cPickle OK
30 - import cProfile ERROR: No module named _lsprof
31 - import cStringIO OK
32 - import calendar OK
33 - import cgi OK
34 - import cgitb OK
35 - import chunk OK
36 - import cmd OK
37 - import code OK
38 - import codecs OK
39 - import codeop OK
40 - import collections OK
41 - import colorsys OK
42 - import commands OK
43 - import compileall OK
44 - import compiler ERROR: No module named compiler
45 - import contextlib OK
46 - import cookielib ERROR: No module named _socket
47 - import copy OK
48 - import csv OK
49 - import datetime OK
50 - import dbhash ERROR: No module named dbhash
51 - import dbm ERROR: No module named ctypes
52 - import decimal OK
53 - import difflib OK
54 - import dircache OK
55 - import dis OK
56 - import distutils OK
57 - import doctest ERROR: No module named signal
58 - import dumbdbm OK
59 - import email OK
60 - import encodings OK
61 - import filecmp OK
62 - import fileinput OK
63 - import fnmatch OK
64 - import formatter OK
65 - import fpformat OK
66 - import fractions OK
67 - import ftplib ERROR: No module named _socket
68 - import functools OK
69 - import gdbm ERROR: No module named cffi
70 - import genericpath OK
71 - import getopt OK
72 - import getpass OK
73 - import gettext OK
74 - import glob OK
75 - import greenlet ERROR: No module named _continuation
76 - import grp ERROR: No module named ctypes
77 - import gzip ERROR: No module named zlib
78 - import hashlib OK
79 - import heapq OK
80 - import hmac OK
81 - import htmlentitydefs OK
82 - import htmllib OK
83 - import httplib ERROR: No module named _socket
84 - import ihooks OK
85 - import imaplib ERROR: No module named _socket
86 - import imghdr OK
87 - import importlib OK
88 - import imputil OK
89 - import inspect OK
90 - import io OK
91 - import json ERROR: No module named json
92 - import keyword OK
93 - import lib2to3 ERROR: No module named lib2to3
94 - import linecache OK
95 - import locale OK
96 - import logging OK
97 - import macurl2path ERROR: No module named _socket
98 - import mailbox ERROR: No module named _socket
99 - import mailcap OK
100 - import markupbase OK
101 - import marshal OK
102 - import md5 OK
103 - import mhlib OK
104 - import mimetools OK
105 - import mimetypes ERROR: No module named _socket
106 - import mimify OK
107 - import modulefinder OK
108 - import multifile OK
109 - import mutex OK
110 - import netrc ERROR: No module named ctypes_support
111 - import new OK
112 - import nntplib ERROR: No module named _socket
113 - import nturl2path OK
114 - import numbers OK
115 - import opcode OK
116 - import optparse OK
117 - import os OK
118 - import pdb OK
119 - import pickle OK
120 - import pickletools OK
121 - import pipes OK
122 - import pkgutil OK
123 - import platform OK
124 - import plistlib OK
125 - import popen2 OK
126 - import poplib ERROR: No module named _socket
127 - import posixfile OK
128 - import posixpath OK
129 - import pprint OK
130 - import profile OK
131 - import pstats OK
132 - import pty ERROR: No module named select
133 - import pwd ERROR: No module named ctypes_support
134 - import pyclbr OK
135 - import pydoc OK
136 - import pyrepl ERROR: No module named pyrepl
137 - import quopri OK
138 - import random OK
139 - import re OK
140 - import repr OK
141 - import resource ERROR: No module named _resource_32_
142 - import rexec OK
143 - import rfc822 OK
144 - import rlcompleter ERROR: No module named __main__
145 - import robotparser ERROR: No module named _socket
146 - import runpy OK
147 - import sched OK
148 - import sets OK
149 - import sgmllib OK
150 - import sha OK
151 - import shelve OK
152 - import shlex OK
153 - import shutil OK
154 - import site ERROR: No module named _socket
155 - import smtpd ERROR: No module named _socket
156 - import smtplib ERROR: No module named _socket
157 - import sndhdr OK
158 - import socket ERROR: No module named _socket
159 - import sre OK
160 - import stackless ERROR: No module named _continuation
161 - import stat OK
162 - import statvfs OK
163 - import string OK
164 - import stringold OK
165 - import stringprep OK
166 - import struct OK
167 - import symbol OK
168 - import sysconfig OK
169 - import syslog ERROR: No module named cffi
170 - import tabnanny OK
171 - import tarfile OK
172 - import telnetlib ERROR: No module named _socket
173 - import tempfile OK
174 - import test OK
175 - import textwrap OK
176 - import this The Zen of Python, by Tim Peters
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
OK
177 - import timeit OK
178 - import toaiff OK
179 - import token OK
180 - import tokenize OK
181 - import tputil OK
182 - import trace OK
183 - import traceback OK
184 - import tty ERROR: No module named termios
185 - import types OK
186 - import unittest ERROR: cannot import name 'result'
187 - import urllib ERROR: No module named _socket
188 - import urllib2 ERROR: No module named _socket
189 - import urlparse OK
190 - import user OK
191 - import uu OK
192 - import uuid OK
193 - import warnings OK
194 - import weakref OK
195 - import webbrowser ERROR: No module named subprocess
196 - import whichdb OK
197 - import wsgiref ERROR: No module named wsgiref
198 - import xdrlib OK
199 - import xml OK
200 - import xmllib OK
201 - import xmlrpclib ERROR: No module named _socket
202 - import zipfile OK
150 worked imports - 53 failed imports
What's about this:
I made a white list of the 150 worked imports and the test will fail if one of there can't be import. The test also print a TODO list with failed imports.
Maybe, these information can be also used to generate a list for README ?
I think we should avoid shipping modules that we know don't work. Can you please add any that don't to the list in https://github.com/rfk/pypyjs/blob/master/tools/module_bundler.py#L93? We can reduce the size of the distribution tarball by skipping them (until we get them working of course!) and simplify the test to "everything we ship should be importable".
OK, have i done here: https://github.com/jedie/pypyjs/commit/76f74e2caf3897e4bd3fefb305e4fcb8804868fb
But i don't know how to update index.json with the module_bundler.py ;(
@rfk do you think the signal module could work for pypyjs? for the unittest package but the signals stuff is easily enough removed from it.
do you think the signal module could work for pypyjs
Not well, although we could probably provide some stub functions that make signal-handler setup "work" but never actually trigger a signal.
There's actually something in emscripten for this allready I'll look into how that works. Allthough like I said it's easily enough removed from the modules. Thanks for the reply :smile:
https://github.com/kripken/emscripten/blob/master/src/library_signals.js
In my https://github.com/rfk/pypyjs/pull/96 i have a test about module imports.
Many modules are in file system, but can't be imported, see complete list here: https://gist.github.com/jedie/8f302ddabd9876f5c927/
I used a hackish way to get a list of all existing module files with this:
IMHO some of the existing files, are useless. e.g.: all *Server files like CGIHTTPServer
Any idea what i should do with this test?!? Is there a way to get more information than
No module named FooBar
?Should i only test a few modules? e.g.: module_bundler.BUILTIN_MODULES ?
EDIT: Or should i parse https://github.com/rfk/pypyjs/blob/master/website/js/pypy.js-0.3.0/lib/modules/index.json ?