Inserts unnecessary newlines into setup.py and HISTORY.txt

[kevswims]

I am a new user to zest releaser. When I run the fullrelease command in a project on Windows, the resulting setup.py and HISTORY.txt has extra newlines inserted between the lines.

I have not tried running on Linux or MacOS to see if that changes anything.

Before:

from setuptools import setup

setup(name='ion_python',
    version='0.1.4.dev0',
    description='desc',
    url='www.example.com',
    author='Kevin Lannen',
    author_email='kevin.lannen@example.com',
    packages=['package_name'],
    zip_save=False,
    test_suite='nose.collector',
    tests_require=['nose'])

After:

from setuptools import setup

setup(name='ion_python',

    version='0.1.5.dev0',
    description='desc',

    url='www.example.com',

    author='Kevin Lannen',

    author_email='kevin.lannen@example.com',

    packages=['ion_python'],

    zip_save=False,

    test_suite='nose.collector',

    tests_require=['nose'])

Running it again seems to insert another set of newlines after each of the lines. Is there a setting that I am missing?

[mauritsvanrees]

We are not testing zest.releaser on Windows. We hope that it works, and we try not to break it, but I am not completely surprised, sorry. I am happy to add a fix, but it is hard for me to test.

What is strange here, is that fullrelease edits the setup.py file twice, so in case of problems I would expect two extra newlines. In your case there is only one extra newline. And the line with the version is fine. So I wonder at which point exactly the newlines are introduced.

fullrelease is a combination of three commands that you can run individually as well: prerelease, release, and postrelease. Can you run those three commands separately and report what they do with setup.py? (Well, release does not touch it.)

If prerelease introduces the extra newlines, can you remove them and see if postrelease puts them back anyway?

If you run fullrelease or the individual commands multiple times, does that increase the newlines even more?

[reinout]

@kevswims: the core of the problem might be the difference between unix and windows newlines. On unix it is \n, on windows \r\n. ("newline" and "carriage return").

[kevswims]

Looks like both the prerelease and postrelease commands are causing issues. It is also inserting extra newlines into the HISTORY.txt file which is shown below. When running the commands individually they each insert extra newlines every time they are run.

I ran through the fullrelease process again and I'm seeing it insert two newlines instead of the one I reported earlier.

Thanks for the tip about this not being an issue on Linux. If this isn't a quick fix I can move this to run on a Linux machine in the short term but I would like to get it running on Windows as well since it's my primary dev environment.

0.1.5 (unreleased)
------------------

- Updated README to reflect jenkins setup for publishing
- Add get_conan_package_info function to get conan package name and version

0.1.4 (2018-01-18)
------------------

- Turned off publishing by zest releaser

0.1.3 (2018-01-18)
------------------

- Updated the readme with how to update the changelog

0.1.2 (2018-01-18)
------------------

0.1.1 (2018-01-18)
------------------

0.1.6 (unreleased)
------------------

- Nothing changed yet.

0.1.5 (2018-01-22)

------------------

- Updated README to reflect jenkins setup for publishing

- Add get_conan_package_info function to get conan package name and version

0.1.4 (2018-01-18)

------------------

- Turned off publishing by zest releaser

0.1.3 (2018-01-18)

------------------

- Updated the readme with how to update the changelog

0.1.2 (2018-01-18)

------------------

0.1.1 (2018-01-18)

------------------

[kevswims]

Not sure if this is useful or not but here's the verbose output for prerelease -v

EBUG: Running command: 'c:\\users\\kelannen\\appdata\\local\\programs\\python\\python36\\python.exe setup.py egg_info'
DEBUG: Stderr of running command 'c:\users\kelannen\appdata\local\programs\python\python36\python.exe setup.py egg_info':
c:\users\kelannen\appdata\local\programs\python\python36\lib\distutils\dist.py:261: UserWarning: Unknown distribution option: 'zip_save'
  warnings.warn(msg)

DEBUG: Running command: 'c:\\users\\kelannen\\appdata\\local\\programs\\python\\python36\\python.exe setup.py --name'
DEBUG: Stderr of running command 'c:\users\kelannen\appdata\local\programs\python\python36\python.exe setup.py --name':
c:\users\kelannen\appdata\local\programs\python\python36\lib\distutils\dist.py:261: UserWarning: Unknown distribution option: 'zip_save'
  warnings.warn(msg)

DEBUG: Running command: 'git symbolic-ref --quiet HEAD'
DEBUG: Running command: 'git status --short --untracked-files=no'
DEBUG: Running command: 'c:\\users\\kelannen\\appdata\\local\\programs\\python\\python36\\python.exe setup.py egg_info'
DEBUG: Stderr of running command 'c:\users\kelannen\appdata\local\programs\python\python36\python.exe setup.py egg_info':
c:\users\kelannen\appdata\local\programs\python\python36\lib\distutils\dist.py:261: UserWarning: Unknown distribution option: 'zip_save'
  warnings.warn(msg)

DEBUG: Running command: 'c:\\users\\kelannen\\appdata\\local\\programs\\python\\python36\\python.exe setup.py --version'
DEBUG: Stderr of running command 'c:\users\kelannen\appdata\local\programs\python\python36\python.exe setup.py --version':
c:\users\kelannen\appdata\local\programs\python\python36\lib\distutils\dist.py:261: UserWarning: Unknown distribution option: 'zip_save'
  warnings.warn(msg)

DEBUG: Extracted version: 0.1.7.dev0
DEBUG: Version indicates development: 0.1.7.dev0.
DEBUG: Removing debug indicators: '0.1.7.'
DEBUG: Running command: 'git ls-tree -r HEAD --name-only'
DEBUG: Found HISTORY.txt
DEBUG: Checking HISTORY.txt
DEBUG: Detected encoding of HISTORY.txt with tokenize: utf-8
DEBUG: Found heading: {'line': 0, 'version': '0.1.7', 'date': 'unreleased'}
DEBUG: Found heading: {'line': 6, 'version': '0.1.6', 'date': '2018-01-22'}
DEBUG: Found heading: {'line': 12, 'version': '0.1.5', 'date': '2018-01-22'}
DEBUG: Found heading: {'line': 18, 'version': '0.1.4', 'date': '2018-01-18'}
DEBUG: Found heading: {'line': 23, 'version': '0.1.3', 'date': '2018-01-18'}
DEBUG: Found heading: {'line': 28, 'version': '0.1.2', 'date': '2018-01-18'}
DEBUG: Found heading: {'line': 31, 'version': '0.1.1', 'date': '2018-01-18'}
Changelog entries for version 0.1.7:

0.1.7 (unreleased)
------------------

- Nothing changed yet.

0.1.6 (2018-01-22)
------------------
DEBUG: Running command: 'c:\\users\\kelannen\\appdata\\local\\programs\\python\\python36\\python.exe setup.py egg_info'
DEBUG: Stderr of running command 'c:\users\kelannen\appdata\local\programs\python\python36\python.exe setup.py egg_info':
c:\users\kelannen\appdata\local\programs\python\python36\lib\distutils\dist.py:261: UserWarning: Unknown distribution option: 'zip_save'
  warnings.warn(msg)

DEBUG: Running command: 'c:\\users\\kelannen\\appdata\\local\\programs\\python\\python36\\python.exe setup.py --version'
DEBUG: Stderr of running command 'c:\users\kelannen\appdata\local\programs\python\python36\python.exe setup.py --version':
c:\users\kelannen\appdata\local\programs\python\python36\lib\distutils\dist.py:261: UserWarning: Unknown distribution option: 'zip_save'
  warnings.warn(msg)

DEBUG: Extracted version: 0.1.7.dev0
DEBUG: Version indicates development: 0.1.7.dev0.
DEBUG: Removing debug indicators: '0.1.7.'
Enter version [0.1.7]:
WARNING: Changelog contains "- Nothing changed yet.". Are you sure you want to release? (y/N)? y
DEBUG: Set heading from '0.1.7 (unreleased)\r' to '0.1.7 (2018-01-22)'.
DEBUG: Set line below heading to '------------------\r'
DEBUG: Running command: 'git ls-tree -r HEAD --name-only'
DEBUG: Detected encoding of setup.py with tokenize: utf-8
DEBUG: Matching version line found: "    version='0.1.7.dev0',\r"
INFO: Set setup.py's version to '0.1.7'
INFO: Changed version from 0.1.7.dev0 to 0.1.7
INFO: History file HISTORY.txt updated.
DEBUG: Running command: 'git diff'
INFO: The '['git', 'diff']':

diff --git a/HISTORY.txt b/HISTORY.txt
index e1161a2..19a3650 100644
--- a/HISTORY.txt
+++ b/HISTORY.txt
@@ -1,33 +1,33 @@
-0.1.7 (unreleased)
-------------------
-
-- Nothing changed yet.
-
-
-0.1.6 (2018-01-22)
-------------------
-
-- Nothing changed yet.
-
-
-0.1.5 (2018-01-22)
-------------------
-
-- Updated README to reflect jenkins setup for publishing
-- Add get_conan_package_info function to get conan package name and version
-
-0.1.4 (2018-01-18)
-------------------
-
-- Turned off publishing by zest releaser
-
-0.1.3 (2018-01-18)
-------------------
-
-- Updated the readme with how to update the changelog
-
-0.1.2 (2018-01-18)
-------------------
-
-0.1.1 (2018-01-18)
+0.1.7 (2018-01-22)
+------------------
+
+- Nothing changed yet.
+
+
+0.1.6 (2018-01-22)
+------------------
+
+- Nothing changed yet.
+
+
+0.1.5 (2018-01-22)
+------------------
+
+- Updated README to reflect jenkins setup for publishing
+- Add get_conan_package_info function to get conan package name and version
+
+0.1.4 (2018-01-18)
+------------------
+
+- Turned off publishing by zest releaser
+
+0.1.3 (2018-01-18)
+------------------
+
+- Updated the readme with how to update the changelog
+
+0.1.2 (2018-01-18)
+------------------
+
+0.1.1 (2018-01-18)
 ------------------
\ No newline at end of file
diff --git a/setup.py b/setup.py
index c15538f..8b3bd2d 100644
--- a/setup.py
+++ b/setup.py
@@ -1,13 +1,13 @@
-from setuptools import setup
-
-setup(name='ion_python',
-    version='0.1.7.dev0',
-    description='Helpers for developing within the Keysight Ion Framework for conan and npm postinstall scripts',
-    url='www.example.com',
-    author='Kevin Lannen',
-    author_email='kevin.lannen@keysight.com',
-    packages=['ion_python'],
-    license='(C) Keysight Technologies Inc.',
-    zip_save=False,
-    test_suite='nose.collector',
+from setuptools import setup
+
+setup(name='ion_python',
+    version='0.1.7',
+    description='Helpers for developing within the Keysight Ion Framework for conan and npm postinstall scripts',
+    url='www.example.com',
+    author='Kevin Lannen',
+    author_email='kevin.lannen@keysight.com',
+    packages=['ion_python'],
+    license='(C) Keysight Technologies Inc.',
+    zip_save=False,
+    test_suite='nose.collector',
     tests_require=['nose'])
\ No newline at end of file

OK to commit this (Y/n)?
DEBUG: Running command: "git commit -a -m 'Preparing release 0.1.7'"
INFO: [throwaway/testing cdb374d] Preparing release 0.1.7
 2 files changed, 46 insertions(+), 46 deletions(-)
 rewrite HISTORY.txt (90%)
 rewrite setup.py (87%)

[reinout]

Helpful! The core problem is here:

DEBUG: Set heading from '0.1.7 (unreleased)\r' to '0.1.7 (2018-01-22)'.
DEBUG: Set line below heading to '------------------\r'

The heading line itself is fine, but the line below contains the extra (windows) \r "carriage return".

@mauritsvanrees: I think we can just call .strip() on that heading line?

[reinout]

Hm. The problem is a bit bigger. It applies to all files that we write: the setup.py and the history file.

It is all concentrated in our utils.py: write_text_file() and read_text_file(). It tries to keep track of encodings (and does a pretty good job). But because it has to keep track of encodings, it opens the files in binary mode. Not in universal mode, which would have taken care of the \r\n problem.

@kevswims, what are you using, python2 or python3? Those treat strings differently.

[kevswims]

@reinout I'm using python3. I have not tried to run it with python2 yet.

[mauritsvanrees]

We might be able to detect the line endings: simply do text.count('\r\n') and compare that with the count for \n and \r and write back with the one that has the highest count. Sounds a bit ugly, but might get the job done.

It has to work on Python 2 and 3 in combination with Linux, Mac and Windows. Tricky stuff...

[bbarney213]

Any updates on this? I encountered this issue today as well. It is causing my Flake8 tests to fail in CI. I like zest.releaser otherwise, but this is a fairly big issue since I primarily use Windows.

[reinout]

Hurray, I now also see this very same problem on the laptop of a colleague! So I've got someone to experiment on.

[reinout]

Ah, we're opening files in binary mode (rb) so that we can do all sorts of necessary tricks to properly detect the encoding. We then split the contents on \n. Those lines might still contain the windows \r from its \r\n pair. Upon writing, we write in regular w mode, not wb. So on windows, the \n gets written out as \r\n.

But most of the lines still had their \r sticking out on the right hand side. So that's now \r\r\n!

See this commit, done on windows: https://github.com/nens/hydxlib/commit/962bce89c09caa295ec2210b83822662766a140b#diff-2eeaed663bd0d25b7e608891384b7298

Note, btw, how the injected new heading line doesn't have the extra \r as it is a fresh line:

https://github.com/nens/hydxlib/commit/962bce89c09caa295ec2210b83822662766a140b#diff-db23dcd814354c954091a9b90dbfd92aR5

Our best bet is probably to right-strip the \r.

[reinout]

I've got a fix in #304

[reinout]

@kevswims, @bbarney213: I've released 6.16.0 with a fix for this issue. So it ought to work properly from now on :-)