Open kloczek opened 2 years ago
TOtally honest here, I don't know much about rpm packages.
How do you get the new release? Through pip right? Do you clone the repo, or use the official release through pypi?
That issue has nothing to do with rpm. You can reproduce that using oprocedure which I've described. Just plese run pytest.
Ohh. The reason for it, is because examples, documentation files and tests has been removed from the pypi release from version 1.8, because of some conflicting names, when installing
I'm not using pypu sdist but tar atogenerated from git tag. https://github.com/NicklasTegner/pypandoc/archive/refs/tags/v1.8.tar.gz
It's because when building, from the pyproject.toml file. In 1.8 we have removed the test and other files.
I would suggest download and extracting the tar.gz, then running the tests, and lastly creating the wheel
I see insise tar ball tests.py.
yes but they aren't included when you build. When the whl gets produced they aren't included.
You can check what is inside autogenerated from git tag tar ball https://github.com/NicklasTegner/pypandoc/tree/v1.8
I know, and in the tarball they are, but my guess is, that when you run the build command, they aren't included, just like when I run python setup.py sdist, because of a change for version 1.8.
My suggestion would be to run pytest before building.
Just tested 1.9 and looks like new two units are failing
I know, and in the tarball they are, but my guess is, that when you run the build command, they aren't included, just like when I run python setup.py sdist, because of a change for version 1.8.
My suggestion would be to run pytest before building.
On typical rpm package build test suite is always executed after builds and install.
So for your errors, it seems that in both instances, it can't find the "python" executable when trying to use a python filter. What do you say would be the best solutions? Trying "python3" before "python", since we actually want py3, or the other way around, where we try python3, if regular "python" executable couldn't be found?
So for your errors, it seems that in both instances, it can't find the "python" executable when trying to use a python filter. What do you say would be the best solutions?
Instead hardcoding "python" executable name use sys.executable
.
Instead hardcoding "python" executable name use sys.executable.
We are not hardcoding the name per say. THe error is from the shibang lines when we test with a python filter.
We are not hardcoding the name per say. THe error is from the shibang lines when we test with a python filter.
Than instead hardcode python executable in shebang line execute script as sys.executable
param.
I've added tree commits to my build procedure:
Patch: %{VCS}/commit/b5565358.patch#/%{name}-Updated-readme-with-correct-batches.patch
Patch: %{VCS}/commit/3e7676dd.patch#/%{name}-Fixed-sort-files-before-processing-292-301.patch
Patch: %{VCS}/commit/b2738b45.patch#/%{name}-Fixes-test-cases-that-uses-python-while-only-python3.patch
and looks like issue still is present <details
+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pypandoc-1.9-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pypandoc-1.9-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra tests.py --deselect tests.py::TestPypandoc::test_pdf_conversion
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.14, pytest-7.1.3, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/pypandoc-1.9
collected 40 items / 1 deselected / 39 selected
tests.py ..........................FF........... [100%]
================================================================================= FAILURES =================================================================================
_____________________________________________________________ TestPypandoc.test_conversion_with_mixed_filters ______________________________________________________________
self = <tests.TestPypandoc testMethod=test_conversion_with_mixed_filters>
def test_conversion_with_mixed_filters(self):
markdown_source = "-0-"
lua = """\
function Para(elem)
return pandoc.Para(elem.content .. {{"{0}-"}})
end
"""
lua = textwrap.dedent(lua)
python = """\
#!{0}
from pandocfilters import toJSONFilter, Para, Str
def func(key, value, format, meta):
if key == "Para":
return Para(value + [Str("{0}-")])
if __name__ == "__main__":
toJSONFilter(func)
"""
python = textwrap.dedent(python)
python.format(sys.executable)
with closed_tempfile(".lua", lua.format(1)) as temp1, closed_tempfile(".py", python.format(2)) as temp2:
with closed_tempfile(".lua", lua.format(3)) as temp3, closed_tempfile(".py", python.format(4)) as temp4:
> output = pypandoc.convert_text(
markdown_source, to="html", format="md", outputfile=None, filters=[temp1, temp2, temp3, temp4]
).strip()
tests.py:384:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pypandoc/__init__.py:93: in convert_text
return _convert_input(source, format, 'string', to, extra_args=extra_args,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
source = b'-0-', format = 'markdown', input_type = 'string', to = 'html', extra_args = (), outputfile = None
filters = ['/tmp/tmpbv7qc_dw.lua', '/tmp/tmpi8dvhe90.py', '/tmp/tmpf8udfv4c.lua', '/tmp/tmpb_8hsrr8.py'], verify_format = True, sandbox = True, cworkdir = None
def _convert_input(source, format, input_type, to, extra_args=(),
outputfile=None, filters=None, verify_format=True,
sandbox=True, cworkdir=None):
_check_log_handler()
_ensure_pandoc_path()
if verify_format:
format, to = _validate_formats(format, to, outputfile)
else:
format = normalize_format(format)
to = normalize_format(to)
string_input = input_type == 'string'
if not string_input:
if isinstance(source, str):
input_file = [source]
else:
input_file = source
else:
input_file = []
input_file = sorted(input_file)
args = [__pandoc_path, '--from=' + format]
args.append('--to=' + to)
args += input_file
if outputfile:
args.append("--output=" + str(outputfile))
if sandbox:
if ensure_pandoc_minimal_version(2,15): # sandbox was introduced in pandoc 2.15, so only add if we are using 2.15 or above.
args.append("--sandbox")
args.extend(extra_args)
# adds the proper filter syntax for each item in the filters list
if filters is not None:
if isinstance(filters, string_types):
filters = filters.split()
f = ['--lua-filter=' + x if x.endswith(".lua") else '--filter=' + x for x in filters]
args.extend(f)
# To get access to pandoc-citeproc when we use a included copy of pandoc,
# we need to add the pypandoc/files dir to the PATH
new_env = os.environ.copy()
files_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "files")
new_env["PATH"] = new_env.get("PATH", "") + os.pathsep + files_path
creation_flag = 0x08000000 if sys.platform == "win32" else 0 # set creation flag to not open pandoc in new console on windows
old_wd = os.getcwd()
if cworkdir and old_wd != cworkdir:
os.chdir(cworkdir)
p = subprocess.Popen(
args,
stdin=subprocess.PIPE if string_input else None,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
env=new_env,
creationflags=creation_flag)
if cworkdir is not None:
os.chdir(old_wd)
# something else than 'None' indicates that the process already terminated
if not (p.returncode is None):
raise RuntimeError(
'Pandoc died with exitcode "%s" before receiving input: %s' % (p.returncode,
p.stderr.read())
)
if string_input:
try:
source = cast_bytes(source, encoding='utf-8')
except (UnicodeDecodeError, UnicodeEncodeError):
# assume that it is already a utf-8 encoded string
pass
try:
stdout, stderr = p.communicate(source if string_input else None)
except OSError:
# this is happening only on Py2.6 when pandoc dies before reading all
# the input. We treat that the same as when we exit with an error...
raise RuntimeError('Pandoc died with exitcode "%s" during conversion.' % (p.returncode))
try:
stdout = stdout.decode('utf-8')
except UnicodeDecodeError:
# this shouldn't happen: pandoc more or less guarantees that the output is utf-8!
raise RuntimeError('Pandoc output was not utf-8.')
try:
stderr = stderr.decode('utf-8')
except UnicodeDecodeError:
# this shouldn't happen: pandoc more or less guarantees that the output is utf-8!
raise RuntimeError('Pandoc output was not utf-8.')
# check that pandoc returned successfully
if p.returncode != 0:
> raise RuntimeError(
'Pandoc died with exitcode "%s" during conversion: %s' % (p.returncode, stderr)
)
E RuntimeError: Pandoc died with exitcode "83" during conversion: Error running filter /tmp/tmpi8dvhe90.py:
E Could not find executable python
pypandoc/__init__.py:420: RuntimeError
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
/home/tkloczko
_____________________________________________________________ TestPypandoc.test_conversion_with_python_filter ______________________________________________________________
self = <tests.TestPypandoc testMethod=test_conversion_with_python_filter>
def test_conversion_with_python_filter(self):
markdown_source = "**Here comes the content.**"
python_source = '''\
#!{0}
"""
Pandoc filter to convert all regular text to uppercase.
Code, link URLs, etc. are not affected.
"""
from pandocfilters import toJSONFilter, Str
def caps(key, value, format, meta):
if key == 'Str':
return Str(value.upper())
if __name__ == "__main__":
toJSONFilter(caps)
'''
python_source = textwrap.dedent(python_source)
python_source.format(sys.executable)
with closed_tempfile(".py", python_source) as tempfile:
> output = pypandoc.convert_text(
markdown_source, to='html', format='md', outputfile=None, filters=tempfile
).strip()
tests.py:334:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pypandoc/__init__.py:93: in convert_text
return _convert_input(source, format, 'string', to, extra_args=extra_args,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
source = b'**Here comes the content.**', format = 'markdown', input_type = 'string', to = 'html', extra_args = (), outputfile = None, filters = ['/tmp/tmp2o_r5jt_.py']
verify_format = True, sandbox = True, cworkdir = None
def _convert_input(source, format, input_type, to, extra_args=(),
outputfile=None, filters=None, verify_format=True,
sandbox=True, cworkdir=None):
_check_log_handler()
_ensure_pandoc_path()
if verify_format:
format, to = _validate_formats(format, to, outputfile)
else:
format = normalize_format(format)
to = normalize_format(to)
string_input = input_type == 'string'
if not string_input:
if isinstance(source, str):
input_file = [source]
else:
input_file = source
else:
input_file = []
input_file = sorted(input_file)
args = [__pandoc_path, '--from=' + format]
args.append('--to=' + to)
args += input_file
if outputfile:
args.append("--output=" + str(outputfile))
if sandbox:
if ensure_pandoc_minimal_version(2,15): # sandbox was introduced in pandoc 2.15, so only add if we are using 2.15 or above.
args.append("--sandbox")
args.extend(extra_args)
# adds the proper filter syntax for each item in the filters list
if filters is not None:
if isinstance(filters, string_types):
filters = filters.split()
f = ['--lua-filter=' + x if x.endswith(".lua") else '--filter=' + x for x in filters]
args.extend(f)
# To get access to pandoc-citeproc when we use a included copy of pandoc,
# we need to add the pypandoc/files dir to the PATH
new_env = os.environ.copy()
files_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "files")
new_env["PATH"] = new_env.get("PATH", "") + os.pathsep + files_path
creation_flag = 0x08000000 if sys.platform == "win32" else 0 # set creation flag to not open pandoc in new console on windows
old_wd = os.getcwd()
if cworkdir and old_wd != cworkdir:
os.chdir(cworkdir)
p = subprocess.Popen(
args,
stdin=subprocess.PIPE if string_input else None,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
env=new_env,
creationflags=creation_flag)
if cworkdir is not None:
os.chdir(old_wd)
# something else than 'None' indicates that the process already terminated
if not (p.returncode is None):
raise RuntimeError(
'Pandoc died with exitcode "%s" before receiving input: %s' % (p.returncode,
p.stderr.read())
)
if string_input:
try:
source = cast_bytes(source, encoding='utf-8')
except (UnicodeDecodeError, UnicodeEncodeError):
# assume that it is already a utf-8 encoded string
pass
try:
stdout, stderr = p.communicate(source if string_input else None)
except OSError:
# this is happening only on Py2.6 when pandoc dies before reading all
# the input. We treat that the same as when we exit with an error...
raise RuntimeError('Pandoc died with exitcode "%s" during conversion.' % (p.returncode))
try:
stdout = stdout.decode('utf-8')
except UnicodeDecodeError:
# this shouldn't happen: pandoc more or less guarantees that the output is utf-8!
raise RuntimeError('Pandoc output was not utf-8.')
try:
stderr = stderr.decode('utf-8')
except UnicodeDecodeError:
# this shouldn't happen: pandoc more or less guarantees that the output is utf-8!
raise RuntimeError('Pandoc output was not utf-8.')
# check that pandoc returned successfully
if p.returncode != 0:
> raise RuntimeError(
'Pandoc died with exitcode "%s" during conversion: %s' % (p.returncode, stderr)
)
E RuntimeError: Pandoc died with exitcode "83" during conversion: Error running filter /tmp/tmp2o_r5jt_.py:
E Could not find executable python
pypandoc/__init__.py:420: RuntimeError
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
/home/tkloczko
============================================================================= warnings summary =============================================================================
pypandoc/pandoc_download.py:62
/home/tkloczko/rpmbuild/BUILD/pypandoc-1.9/pypandoc/pandoc_download.py:62: DeprecationWarning: invalid escape sequence \.
regex = re.compile(r"/jgm/pandoc/releases/download/.*(?:"+processor_architecture+"|x86|mac).*\.(?:msi|deb|pkg)")
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================================= short test summary info ==========================================================================
FAILED tests.py::TestPypandoc::test_conversion_with_mixed_filters - RuntimeError: Pandoc died with exitcode "83" during conversion: Error running filter /tmp/tmpi8dvhe90...
FAILED tests.py::TestPypandoc::test_conversion_with_python_filter - RuntimeError: Pandoc died with exitcode "83" during conversion: Error running filter /tmp/tmp2o_r5jt_...
========================================================== 2 failed, 37 passed, 1 deselected, 1 warning in 4.93s ===========================================================
Additionally after --deselect failing units I see some warnings:
+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pypandoc-1.9-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pypandoc-1.9-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra tests.py --deselect tests.py::TestPypandoc::test_pdf_conversion --deselect tests.py::TestPypandoc::test_conversion_with_mixed_filters --deselect tests.py::TestPypandoc::test_conversion_with_python_filter
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.14, pytest-7.1.3, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/pypandoc-1.9
collected 40 items / 3 deselected / 37 selected
tests.py ..................................... [100%]
============================================================================= warnings summary =============================================================================
pypandoc/pandoc_download.py:62
/home/tkloczko/rpmbuild/BUILD/pypandoc-1.9/pypandoc/pandoc_download.py:62: DeprecationWarning: invalid escape sequence \.
regex = re.compile(r"/jgm/pandoc/releases/download/.*(?:"+processor_architecture+"|x86|mac).*\.(?:msi|deb|pkg)")
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=============================================================== 37 passed, 3 deselected, 1 warning in 4.33s ================================================================
@kloczek can you try the latest development snapshot, that should fix the failing test cases.
After replace last patch with:
Patch: %{VCS}/commit/b5565358.patch#/%{name}-Updated-readme-with-correct-batches.patch
Patch: %{VCS}/commit/3e7676dd.patch#/%{name}-Fixed-sort-files-before-processing-292-301.patch
Patch: https://github.com/JessicaTegner/pypandoc/commit/b2738b45.patch#/%{name}-Fixes-test-cases-that-uses-python-while-only-python3-is-available.patch
pytest still fails ..
well we are using sys.executable to run the python tests now, so I don't see how it could fail...
Just retested 1.11 and still I see three units failing
@kloczek From the 2 last ones:
python_source = '''\
#!{0}
"""
Pandoc filter to convert all regular text to uppercase.
Code, link URLs, etc. are not affected.
"""
from pandocfilters import toJSONFilter, Str
def caps(key, value, format, meta):
if key == 'Str':
return Str(value.upper())
if __name__ == "__main__":
toJSONFilter(caps)
'''
python_source = textwrap.dedent(python_source)
python_source.format(sys.executable)
We are setting the shebang line by using "sys.executable", so only reason why it can't run the python filters would be because either the "sys.executable" is incorrect, or is still set to regular python (somehow).
Can you try running something like the following, to check what the sys.executable is set to when running our tests?
import sys
print(sys.executable)
For the first one, the one about the data files. That's an error in the test case, where "sandbox" is specifically set to True, even though the default is now False. THat should be easy enough to fix, by just omitting the sandbox parameter all together
[tkloczko@pers-jacek SPECS]$ python3
Python 3.8.16 (default, Jan 30 2023, 13:00:00)
[GCC 13.0.1 20230127 (Red Hat 13.0.1-0)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.executable)
/usr/bin/python3
>>>
@kloczek can you test master now, after the work pr pr #328 the python ones should be fixed
This issue should be able to be closed now. ping @kloczek
Hmm .. just retested 1.13 and pytest still fails in 3 units 🤔
@kloczek Can you try the following:
I'm trying to package your module as an rpm package. So I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.
python3 -sBm build -w --no-isolation
build
with--no-isolation
I'm using during all processes only locally installed modulesHere is pytest output: