sergeevabc commented 7 years ago

What is the reason to supply command-line pandoc.exe within MSI installer instead of ZIP? Current approach clutters up Windows system, so I call to reduce the footprint. Until then workaround is

mkdir "%TEMP%\pandoc\"
start /wait msiexec.exe /a pandoc-1.19.1-windows.msi /qn targetdir="%TEMP%\pandoc\"
xcopy /y "%TEMP%\pandoc\Pandoc\pandoc.exe" C:\Utils\Console\
rmdir /s /q "%TEMP%\pandoc\"

wilx commented 7 years ago

I guess that supplying plain ZIP distribution might be a good idea and useful for some users. However, I do not see the point of you doing what you do with that. Why can't you just install either globally or locally to the directory that you want?

MSI distribution should stay, IMHO. It is useful in that the installer installs and reinstalls the executable for me automatically. If you use the MSI then you can install it either globally or locally for your single Windows user.

Also, Google search seems to mention that MSI files can be extracted using 7-Zip. I have not tried though.

sergeevabc commented 7 years ago

First, let us recall…

I do not see the point of you doing what you do with that. Why can't you just install…

Installer suggests software is not portable, whereas Pandoc is portable since you can launch it from any folder with no traces left behind, i.e. it does not pollute filesystem, start menu and registry. However installer does. So instead I unpack app to some folder mentioned in PATH and update it likewise. When app is supplied in ZIP, it is as simple as drag’n’drop (with no tricky aforesaid unpacking routine).

With MSI you can install it either globally or locally

It makes sense when app stores settings per user, whereas pandoc.exe has no per user settings.

Google seems to mention MSI files can be extracted using 7-Zip.

It can, but you need no 3rd-party app to extract MSI as there is native msiexec.exe (see above).

jgm commented 7 years ago

One thing the MSI does is change the user's path so that it includes the directory pandoc goes in.

Installing from the MSI is also easier for people who don't know much; they don't have to know which directories are in their paths, or even what a directory is.

(Of course, pandoc might not be much use to such users.)

sergeevabc commented 7 years ago

@jgm, you did a great job composing the comprehensive Pandoc’s manual. Let’s look through it again — do you truly believe the one who deals with such complexity of command-line switches inside dark cold terminal window with small-sized font is not aware that any CLI app works globally only after its folder is added to PATH either via GUI or setx path?

Getting started section even teaches about command cd to get around the piles of folders, but omits to mention the importance of altering PATH beforehand since it’s done automatically here. Thus, newcomers are corrupted — a false premise is cultivated that other CLI apps would work globally as well. For example, iconv -t utf-8 input.txt | pandoc would fail because this iconv is not added to PATH automatically.

Deep down you agree, hence your last comment in round brackets.

jgm commented 7 years ago

A zip would be a lot simpler, certainly.

I'd like to get more opinions about this. Ideally it should be raised on pandoc-discuss, where more people will see it. Perhaps @sheremetyev, who got me started using an msi, would like to chime in here.

mrowa commented 7 years ago

I'm a fresh pandoc user and I was happy I could install it from msi - rather than having to keep my references in absolute paths or than adding it by hand to some directory in $PATH (and if you don't have your "tool" dir in PATH, you need to relog after adding a new one). Adding zip distribution might make life easier for some, but proper installer is a feature - please don't remove it for those who actually find it useful.

sergeevabc commented 7 years ago

@jgm, I believe the case of @mrowa is exactly how newcomers are corrupted by MSI at present.

In the short run they are happy to get Pandoc in PATH out of the box, whereas in the long run it teaches them little of how CLI works and what to expect. However the ~drug-like addiction~ expectation of other CLI apps to be available the same way is cultivated. But developers fulfill it rarely, why? One reason is that PATH variable is limited, separate folders of all globally required apps would not fit in. Thus, marvelous gems like iconv, grep, imagemagick, ffmpeg, sysinternals, etc remain terra incognita for “corrupted users”.

Good practice is to keep CLI apps together and add that very folder to PATH via GUI or setx once.

mrowa commented 7 years ago

On any other platform, installed CLI tool by default is already in the PATH-equivalent. Node's npm and Python's pip let you install scripts/software in their own PATH-ed directories.

@sergeevabc please don't force me to micro-manage my software, even if it's your choice to do so. There is no "Best practice" like this for everyone. It's as solution if you're already having a problem with too long PATH. And default add-to-PATH it's not "Corrupting users", it's good UX.

Actually, PATH is just part of the MSI's features, others being:

easy update
good default install locations, be it user or system-wide
just works out of the box
standarized for enterprise use
you don't have to use console if you don't want too - pandoc works fine with task runners

sergeevabc commented 7 years ago

@mrowa, this issue and subsequent comments are related to Windows, not any other platform. Node and Python are not just utilities, but runtime environments hence the complex integration. Practice of keeping utilities together goes back decades, update is as easy as unpack to the same folder. Let alone one root folder is easier to navigate to, backup and transfer to thumbdrive or another location. As for lobbying enterprise norms, let me remind you smoking was advocated by celebs and doctors once.

Major point: Pandoc is a CLI utility and should be distributed as such w/o pretending to be more than that.

sergiocorreia commented 7 years ago

Just to chime in. I understand why @sergeevabc wants a zip file, but for a normal user MSI files are incredibly useful.

I have both sysinternals and ffmpeg installed, and I hate that those programs just ship as ZIP files (so I have to hand hold them: extract the zip files, place them in a specific folder, add that folder to PATH, etc.).

sergeevabc commented 7 years ago

@sergiocorreia, CLI apps are not meant for “normal users” by design, or better to say not for “indolent aka don’t make me think users”, since such fat-free flexibility requires a certain level of commitment and knowledge (e.g. how to navigate, chain commands, route output). Setting environment variables (like PATH or OPENSSL_CONF for OpenSSL) is a crucial part of that knowledge and is done once and for all soon after installing Windows.

You hate the way most CLI apps are distributed because deep down you expect integration level of graphical apps usually supplied with installers (like using Bandizip via context menu versus 7z a archive -t7z -mx -ms=2g -m0=LZMA2:d64m:fb256 -mmt2 *). Therefore you opt to preserve a crumb of point’n’click comfort which comes with MSI, but then you still have to deal with dark cold terminal window.

Instead of pulling Pandoc out of CLI niche, you should ask for GUI wrapper.

crmackay commented 7 years ago

another point:

I have had situations (on VMs etc) where I do not have permissions required to run an .msi and "install" Pandoc. A simple .exe would have been (and still would be) really helpful in this situation. Instead I have to extract the .msi and dig out the .exe, which while possible, is a bit cumbersome.

with respect to the other points raised above: Can't they both exist? an Installer and standalone Executable? Let the user decide?

The installation instructions could default to the installer, and saying something like "If you want and know how, you can use this simple .exe file instead"

jgm commented 7 years ago

+++ Chris MacKay [Dec 22 16 15:04 ]:

Can't they both exist? an Installer and standalone Executable? Let the user decide?

Sure, that's feasible. We're already generating both on appveyor, so this is just a bit more work to upload them both.

ickc commented 7 years ago

@sergeevabc

do you truly believe the one who deals with such complexity of command-line switches inside dark cold terminal window with small-sized font is not aware that any CLI app works globally only after its folder is added to PATH either via GUI or setx path?

No one need to believe it. Just go through pandoc-discuss and see questions asked there and one will see that's a fact.

Some users just uses very basic commands like pandoc -s -o test.tex test.docx without knowing what's going on.

In the short run they are happy to get Pandoc in PATH out of the box, whereas in the long run it teaches them little of how CLI works and what to expect. However the drug-like addiction expectation of other CLI apps to be available the same way is cultivated. But developers fulfill it rarely, why? One reason is that PATH variable is limited, separate folders of all globally required apps would not fit in. Thus, marvelous gems like iconv, grep, imagemagick, ffmpeg, sysinternals, etc remain terra incognita for “corrupted users”.

I agree that to master pandoc one would needs to learn a lot of stuffs, and eventually understands what you are talking about, and what you think they should know. But there's no point to steepen the initial learning curve. They will learn about the complexity involved when they need to. But why forcing them to learn it earlier? It might be in your philosophy that more people just be "educated" in this aspect, but the fact is if you steepen the initial learn curve that they can't even start using it with their current knowledge, they will just pass and not say "let me learn these stuffs because they will be useful and allow me to use pandoc". In the end, who said anyone should master pandoc if not mastering it already serves what they need? ("they/them" refers to common peopl generally, there are always exceptions if you are picky.)

Reading the adjectives you use, you are strongly biased, perhaps also a control freak, perfectionist, or might think yourself better than the others. As a control freak, I think being a control freak is a good thing, but only when you're "controlling yourself", in both sense. And if you want to make the world better (which probably is your intention here, where "better" might be of problematic definition), I wish you will see your way will not works.

IMO, I envisioned in the future, one can use pandoc without having to go through command line. You might think I'm crazy, but note that right now you can finish a LaTeX documents without having to touch command line at all. LaTeX is much more complex that pandoc in a certain way, and yet it is still more accessible than it. IMHO, forcing end-users to learn all those cli stuffs to use pandoc is a wrong direction. They should be there at your disposal, but not mandatory.

Just to make sure: I think options are good, so I'm not opposing having an alternative installation method if it is doable and not too complicated.

sergeevabc commented 7 years ago

@ickc Just go through pandoc-discuss and see questions…

Pandoc has FAQ, which is the very place to accumulate repeated answers. Making MSI was an overkill.

Forcing users to learn CLI stuff to use Pandoc is a wrong direction…

Pandoc forces no more than any other CLI gem like Wget, Jdupes or Rhash. No pain no gain.

Reading the adjectives you use, you are strongly biased…

2+2=4. I’m that biased, yes. But I’m also aware it happens the other way sometimes.

@crmackay, tell us as a honest fellow of the medical society whether doctors’ schedules are overloaded mainly because of patients’ wrong choices? E.g. disorders as a result of excessive sugar consumption. It is reminiscent of the security issue of “dancing pigs” when user chooses amusement over danger warning. We (who have learned to foresee a little) should be wiser than soda manufacturers (who supply sweet poison with different flavours to “let the user decide”) and above all stick to the good old maxim “do not harm” from the Hippocratic oath, even while developing and sharing software.

That’s why we should strangle the corruption of newcomers at birth by a) replacing MSI with ZIP and b) adding a paragraph about PATH to the user’s guide, where accessory CLI stuff is mentioned anyway. But since @jgm seems to treat any arguments equally, even voiced by “soda lovers”, and is ready to shoulder a double burden, then MSI + ZIP is a feasible compromise indeed.

ickc commented 7 years ago

@sergeevabc

Pandoc has FAQ, which is the very place to accumulate repeated answers. Making MSI was an overkill.

You are idealistic, not realistic. There's a whole bunch of people that just don't RTM. And even when I first learn using pandoc after RTM wholly, I still made stupid mistakes and asked stupid questions. Again, there's no point to increase the initial learning curve.

Pandoc forces no more than any other CLI gem like Wget, Jdupes or Rhash. No pain no gain.

Do you realize there's some other apps has pandoc as a dependency, and the end users are required to install pandoc, but the said app do not requires CLI?

2+2=4. I’m that biased, yes. But I’m also aware it happens the other way sometimes.

You bias towards reason/logic only. Again, I enjoy being logical, but just like the "control freak" argument, it is only nice when it is enforced on one-self, but not on others. After all, you are not everyone's teacher/boss.

ickc commented 7 years ago

@sergeevabc

Just to make it clear: I don't think there's a need for any argument against your points. Because obviously the way you put it make you losing your argument.

The reason I joint the conversation is to plead you as a fellow human. I assume you want to make the world better, and if that's true, you need to understand what's better is "subjective". Logical person is very good at objective things, like 2+2=4, but may/can be poor on "subjective" matters.

Sometimes we need to give up our own pursue of perfection, and try to understand not everyone is like you (or as you argued, the majority of pandoc users). If we can indeed make pandoc perfect (and doing so "objectively"), but in the end less appealing and hence lesser user adoption, what's the point of it?

@jgm is a philosophical professor, one can guess he is a very logical person, and might see a lot of problems in the softwares/cli/users/etc. in ways we don't realize. But as far as I can see, he is willing to make compromise to suit the community better. Being able to compromise, being selfless, being able to put down one's own pursue of perfection are the qualities we all need to learn from him. (Just my extrapolation on him though.)

And if you have these qualities, I am sure you will be more effective to make the world better. After all, your suggestion of having zip has been accepted. The problem lies in the way you push it, and your view that "compromise" of having msi+zip is a bad thing. After all, not every people being "taught" by you are as nice, you know.

sergeevabc commented 7 years ago

@ickc You need to understand what’s better is subjective…

Turning technical debate into a sermon on moral relativism is somehow inappropriate to what @jgm invited us to discuss, let alone I have enough gray hair to challenge such opinion since universal truths like “if prick one with a pin, one bleeds” exist. Gluttony, for example, apart from religious reason, is evil as it causes disorders thus objectively weakens the society, so it’s better to maintain a moderate lifestyle from inner to outer world like DiCaprio does.

Problem lies in the way you push it…

The reason I’m passionate (hence the scrutiny of feedback) is not the struggle for perfection or own benefit, but the long-term outcome. Would you add ASCII version of Rita Hayworth to Pandoc in the light of dozen requests? Or some ideas should be avoided no matter how appealing they seem to not get into confusion later like “What’s the difference between JavaScript and Java”?

You are idealistic, not realistic…

If anything, I am pragmatic. Should you choose to make app as friendly as possible, GUI is the way to go. It would add context menu to Explorer, checkboxes and dropdowns instead of switches, F1 to view help. MSI installer here is a disservice that cultivates false expectation that other CLI apps would work globally

Do you realize there's some other apps has pandoc as a dependency…

Let those apps manage the dependencies correctly, not by assuming they are available globally. For example, a prominent media player and converter Foobar2000 allows to set path to encoders.

ickc commented 7 years ago

@sergeevabc You're right. And I'm wrong about you. Forget about what I said. You know what I mean.

ickc commented 7 years ago

This is not intended to speak to @sergeevabc, if you happen to read this, please ignore it.

Since the MSI is not getting away, adding a seperate ZIP release for Windows might clutters the releases, and might also complicates the level of involvement per release.

And since those who need a ZIP release would certainly have no difficulties to go through command line, I'm thinking if it is possible to write a script that given a pandoc version, it will automatically download the corresponding MSI file and returns any of the required contents a ZIP distribution provided. Then the said script can be just put in the repository and referred to in relevant sections. I don't script on Windows, but the OP might have already provided part of the said script.

Another question is, pandoc on Mac can be installed through homebrew, and somewhere in the repository it is already recommended as an alternative installation method on Mac. In the past I googled about homebrew-like tools on Windows and there seems a few (but I don't really know the details about it). If a similar alternative exists, it might be much easier to write a "formula" in such tool than to add another package in releases. But on the other hand, that "homebrew alternatives" on Windows might not be as de facto as homebrew on Mac, so a choice on any such tools might be questionable.

Note that homebrew doesn't always mean build from source, if it is why cabal doesn't solve the problem. One can write a brew-cask formula to install packaged binaries too. So I'm referring to this later correspondance on Windows.

sergeevabc commented 7 years ago

Creating ZIP is relatively easy since we compress the same folder which is processed while creating MSI.

Rationale behind ZIP is to get portable app the easiest way without dependencies like Chocolatey or any other third-party package manager, i.e. just open site via browser and download the latest version.

Another way, used by WikidPad and Hexchat, is to supply EXE installer (allows to mitigate permissions issues) with optional “portable mode”, so it could be one solution for all instead of MSI + ZIP.

ickc commented 7 years ago

When reading through #3332, there's a mildly related issue: often time people do not have sudo privilege to install the latest pandoc. So I think whenever possible the Manual/installation page might need to suggest a command to unpack the deb let say to install it locally. i.e. it is similar to this issue as to offer a portable way of installing it.

jgm commented 7 years ago

In other words, we could offer a linux binary in a zip container.

I haven't done this, partly because it's not hard to extract the binary from the deb container; in fact, the INSTALL.md file already contains instructions for how to do this:

    ar p $DEB data.tar.gz | tar xvz --strip-components 2 -C $HOME/.local/

+++ ickc [Jan 05 17 18:42 ]:

When reading through [1]#3332, there's a mildly related issue: often time people do not have sudo privilege to install the latest pandoc. So I think whenever possible the Manual/installation page might need to suggest a command to unpack the deb let say to install it locally. i.e. it is similar to this issue as to offer a portable way of installing it.

ickc commented 7 years ago

I haven't done this, partly because it's not hard to extract the binary from the deb container; in fact, the INSTALL.md file already contains instructions for how to do this:
    ar p $DEB data.tar.gz | tar xvz --strip-components 2 -C $HOME/.local/

Ah, right. I think a .zip is not needed when such command is so simple. I think the "Installation" instructions just needed to point out this is the way to install without sudo privilege. And by the way, right now the Title of http://pandoc.org/installing.html is just <title>Pandoc - </title>. It should be something like "Pandoc - Installing". I can look into the source of your website and make pull request there if you want.

And going back to Windows: the command given in the original post seems to do a similar thing for Windows (to extract the binary from the .msi). As I suggested earlier, may be we should just put this command in the "Installation" page. This way it won't clutter the "Releases". I don't know if the suggestion of replacing .msi with .exe would make this better/easier though.

And lastly, do you know a non-sudo method to install the .pkg for Mac? brew install pandoc does not requires sudo, but only if they have install homebrew beforehand (which requires a lot of sudo). If it can't be done, may be actually it's the Mac platform that need a "portable" version of "Releases" (especially now Mac is getting more and more difficult to install non-App Store apps...).

jgm commented 7 years ago

+++ ickc [Jan 06 17 15:48 ]:

I think the "Installation" instructions just needed to point out this is the way to install without sudo privilege. And by the way, right now

They do. The passage I quoted is right on the page.

the Title of [1]http://pandoc.org/installing.html is just Pandoc</p> <ul> <li>. It should be something like "Pandoc - Installing".

I'll fix this.

I've also added instructions for manually installing executables from both the msi and the osx pkg.

sergeevabc commented 7 years ago

To sum up: instead of providing a CLI tool as a ready-to-use executable with an optional one-line advice on how integrate it further (which is expected way to distribute CLI tools for decades), you force Win and Mac users to look through the docs for tricky instructions on how dig the executable out of superfluous packages first (which is why the issue was created initially). O brave new world… Utterly upside down.

jgm commented 7 years ago

The package also installs documentation and in the case of Macs, man pages. CLI tools ought to have man pages, and it's no fun installing these manually. There's plenty of reason to have a package manager install these. In the Windows case, an HTML version of the User's Guide is installed, as well as a copy of the license, and the path is set. The installer also shows users the license and asks them to accept it.

We are not the only ones who use package installers for Windows CLI tools, either. See http://gnuwin32.sourceforge.net/packages.html for example.

Anyway, I may end up providing a zip as well, since it's easy to do so, but your grandstanding is really unnecessary.

+++ Aleksandr [Jan 07 17 04:59 ]:

To sum up: instead of providing a CLI tool as a ready-to-use executable with an [1]optional advice on how integrate it further (which is [2]expected way to distribute CLI tools for decades), you force Win & Mac users to look through the docs for [3]tricky instructions on how dig the executable out of superfluous packages first.

O brave new world… Utterly upside down.

ickc commented 7 years ago

I experimented with a way to extract all the msi/deb/pkg on a single platform (Ubuntu) and now have a working script on it.

So I can setup a separate repository (say, pandoc-portable) to auto-extract all those packages using Travis. The advantages are:

not cluttering the current pandoc "Releases" page, also means nothing on pandoc repo needed to be touched
older version of pandoc can be processed by Travis in exactly the same way (if you add zip releases, only newer releases has the zips). This can be important for CI and when people want to build against a particular pandoc version.
Since it is using Travis, the releases is almost fully automated. The only manual step needed is after each pandoc releases, add a git tag release in the pandoc-portable repository), and the portable release would be ready in about 2 minutes.

More details:

If one click the link above, one can see the script is really short. The majority of the effort to write it is really to study how each packages can be unpack, and how it can be done on a single platform (in this case, Ubuntu).
Travis isn't setup yet. But I imagine if @jgm approves this, Travis can be setup within 1/2 hour (if nothing goes wrong...). The repository can be transferred under the jgm umbrella, or if a pandoc-org GitHub Organization is setup, under that one. (or the current pandoc-extras GitHub Organization)
I left the .tar.gz as is rather than converting it to .zip, because I think it is quite common on Linux.
The folder structure inside the archives isn't modified yet, nor anything deleted. If we can decided which to keep, and where to put, it can be easily changed. (But I think leaving it "as is" is better.)

sergeevabc commented 7 years ago

@jgm Sir, debate involves challenging arguments, not manners. Being assertive is not a mauvais ton, right? Especially when one challenges status quo, just like the protagonist of Sidney Lumet’s “12 angry men”.

CLI tools ought to have man pages…

Basic help is generally displayed when app is executed w/o arguments or with -h, --help switches. Complete user’s guide is then placed along with the executable in the same ZIP, installer is superfluous.

LAME 64bits 3.99.5 (http://lame.sf.net)
Usage: lame [options] infile outfile

       lame --help        for general usage information
       lame --longhelp    for a complete options list

Installer also shows the license and asks them to accept…

Software license is rarely taken seriously, i.e. skipped. Effective upon use whether is read or not, CLI tools keep license in a separate file or within executable.

flac - Command-line FLAC encoder/decoder version 1.3.1
Copyright (C) 2000-2009  Josh Coalson
Copyright (C) 2011-2014  Xiph.Org Foundation

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

dd for Windows 0.6beta3
Written by John Newbigin <jn@it.swin.edu.au>
This program is covered by terms of the GPL2

We are not the only ones who use package installers… See gnuwin32…

Win ports of UNIX tools depend on Cygwin libraries hence the installers, Pandoc is dependency-free.

I may end up providing a zip as well, since it's easy to do so…

Foregoing richly garnished reasoning calls to act out of long-term vision rather than attitude to please.

jgm commented 7 years ago

+++ Aleksandr [Jan 07 17 13:40 ]:

Sir, debate involves challenging arguments, not manners.

I'm not interested in debating. You've stated your preferences and given your reasons, and I've taken these into account. No decision has been made yet on this matter. But please refrain from rants about what a terrible thing it is that we're not doing things your way. That's the kind of thing that makes volunteer open-source work unpleasant.

I note that you can already get the complete manual from the executable:

pandoc --print-default-data-file MANUAL.txt

This could be made into a --long-help option to make things easier for Windows users, so perhaps that's worth considering.

jgm commented 7 years ago

I'll also note that there are two kinds of users, (a) the kind that knows how to copy an executable into a directory in their path (and knows what a path is), and (b) the kind that doesn't.

Type (b) users are going to be lost if all we provide is a zip, plus some "tricky instructions" on the website for copying and setting a path. On the other hand, type (a) users are precisely the kind of users who will be able to follow the "tricky instructions" for extracting the binary from the MSI.

So, if we only provide one installation option, it seems to me it should be an MSI. The only question is whether we should also provide a zip. Doing so would be simple. I'm already generating the zip in appveyor builds, so I'd just have to upload it to GitHub. Perhaps there'd be some risk that users would be confused about which Windows package to use, I don't know.

ickc commented 7 years ago

Perhaps there'd be some risk that users would be confused about which Windows package to use, I don't know. @jgm

I think it will, because there's always a "least-informed" crowd who download pandoc for the 1st try. And personally I think this crowd is most important to serve — so that they will stay and the community will grow. That's why I am opposing to the idea of having one more release which clutters the releases page.

In addition, as long as we offer a .zip for Windows, people will start to ask the same thing for the other platforms. Since the GitHub releases automatically has 2 archives for source code, the no. of attachments in each release will grow from 5 to 8.

But there are solutions to this problem:

we "abandon" the GitHub release page for newbies. Since we already have Pandoc - Installing pandoc page, which currently point to the GitHub releases, we can link to the msi/deb/pkg directly from pandoc.org instead of redirection.
- This has an added benefit of not having the often very long "release summary" (distracting newbies) before one can scroll down to the bottom and find the downloads.
- A suggestion by @jgm in pandoc-discuss means it would only involve an extra simple script in pandoc-website.
Alternatively, like I suggested earlier, releasing all .zip through a separate repo by Travis. This is totally automated (except the creation of a git tag).
- It has an added benefit of being able to create .zip for older releases almost automatically. This will be helpful for any CI to switch to using .zip (which is easier and probably faster to setup for CI) immediately, which depends on the availability on the older versions of pandoc.
- (This point is not related to a separate repo per se — skip this for brevity.) The extra -1 in the .deb releases, e.g. pandoc-1.19-1-amd64.deb, can be "scripted away" in the .zip too (since the -1 is an artifact of Debian packages). This will shave an extra second for an query, and also simplifies the CI script people need to use. FYI, the Travis' builds image for language R still hardcoded the -1. The 3 guys maintaining that are supposed to be masters of CI. So the fact they made this mistake means any other "laymen" setting up CI using pandoc will very likely to make the same mistake. (I'm tempted to elaborate on the importance of the easiness of setting up pandoc in CI, along the line of "reproducible research"...)

Although the 2 suggested solutions are alternative solutions, both can be implemented together to solve a different aspects of the problem (1 for newbies, 2 for experienced users / CI).

ickc commented 7 years ago

I opened pandoc-extras/pandoc-portable to use travis build to deliver alternative archives. You can see the results in https://github.com/pandoc-extras/pandoc-portable/releases/tag/1.19.1.

To add a new build is as simple as creating a git tag and let travis does its magic. If @jgm has no objection, I plan to create git tag as far back to about 1.15 when .deb package first released. I guess this will be helpful for CI that requires speed and availability of multiple versions.

jgm commented 7 years ago

+++ ickc [Jan 16 17 07:09 ]:

I opened pandoc-extras/pandoc-portable to use travis build to deliver alternative archives. You can see the results in [1]https://github.com/pandoc-extras/pandoc-portable/releases/tag/1.19.1 .

To add a new build is as simple as creating a git tag and let travis so its magic. If [2]@jgm has no objection, I plan to create git tag as far back to about 1.15 when .deb package first released. I guess this will be helpful for CI that requires speed and availability of multiple versions.

Actually I'm not too keen on having multiple distribution points of releases that look "official," as these will with the pandoc-extras domain. Installing a binary is always a potential security risk, and having a single canonical place to get the pandoc binaries is a good idea, in my opinion.

If there's a need for zips, I'd rather provide them right on the canonical release page.

The main argument against doing this is that it might create confusion about what to install -- but having a separate place with binaries would do that too.

ickc commented 7 years ago

potential security risk

Ah, yes, I thought about that too. I was going to add in the README there that everything is in the open including the travis build history. So one can see that all the binaries is coming from https://github.com/jgm/pandoc.

On the other hand, if GitHub organization pandoc-org will be setup, I think pandoc-portable will fall in that better than pandoc-extras. Or else just transfer it under the jgm umbrella.

One advantage of having pandoc-portable is it is independent of the jgm/pandoc repository. So any potential things added there will not affects/complicates the jgm/pandoc setup, whether it is Travis/Appveyor/Makefile, etc. Potentially in the future, alternative builds can be provided by pandoc-portable (or its new name). e.g. 32-bit on Linux, ARM on Linux, etc. (which I know there's a need for, albeit maybe a small one. Personally I think ARM is going to be increasingly important because in consumer space, ARM computers might be more that x86/x64 and is certainly much cheaper. Sadly GPLv2 softwares cannot enter the App Store will might hinder pandoc's growth.)

Having 3rd parties builds are quite common, examples are p7zip (porting to Unix) and KeePass (which has distinct sections on "Contributed/Unofficial KeePass Ports" and "Contributed/Unofficial KeePass Packages"). And there already are 3rd party build for pandoc (e.g. from Pandoc - Installing pandoc, or even something like pypandoc which includes the binaries of pandoc). So in a sense, pandoc-portable might even be improved to the extent that it grabs all these alternative binaries and put it in one umbrella (not that it is an easy task, nor saying I'm doing it now).

The main argument against doing this is that it might create confusion about what to install -- but having a separate place with binaries would do that too.

Like the other 3rd parties builds, a mentioning in Pandoc - Installing pandoc and an emphasis that it is a 3rd party mirror will be sufficient. I think the main pandoc releases should focus on the majority of the mass, but not all of them. It is these other 3rd parties builds/mirrors that fulfill the needs of the niche (however small but potentially important one). I think the need of portable zip is in the niche (else we would have heard about the complaint a lot), whether it is for CI, for non-sudo, for fun (this thread is certainly funny), etc.

And I think Pandoc - Installing pandoc will always be the canonical release page. It is going to be very difficult for people to stumble upon pandoc-extras/pandoc-portable and thought that was official (especially when the README page made this very clear). And by the way, for this reason I suggested to link to the latest .deb|.pkg|.msi files directly on Pandoc - Installing pandoc rather that redirecting them to GitHub's release page. From a newbies perspective, they will be confused about pandoc.org vs github.com/jgm/pandoc.

ickc commented 7 years ago

Another related thing is nightly build. I personally have experienced and seen bug reports on the latest pandoc that is fixed in the master branch. Building from source to check before every bug report is impractical for some and impossible for most. But if a nightly build is provided, then one can redirect people to install and report on their own.

It wouldn't fall under pandoc-portable. It seems doable by Travis (may even auto deploy daily), but no idea about appveyor.

ickc commented 7 years ago

As a demo, here's the nightly build for osx and linux: https://github.com/pandoc-extras/pandoc-nightly/releases

It only contains the pandoc executables, as it is just a nightly build (didn't write up readme and license yet)

jgm commented 7 years ago

+++ ickc [Jan 16 17 17:26 ]:

It wouldn't fall under pandoc-portable. It seems doable by Travis (may even auto deploy daily), but no idea about appveyor.

we already generate MSI and zip for each build on appveyor. (see under Artifacts)

ickc commented 7 years ago

we already generate MSI and zip for each build on appveyor. (see under Artifacts)

Nice! Never knew this. Good to know since I don't know anything about Windows scripting and appveyor.

I searched around and see that Travis also support artifacts, but it seems it requires S3. Unless you want to use the artifacts via S3, I can add a cron job to pandoc-nightly to auto deploy daily (where each files will be labeled by the commit hash in pandoc).

ickc commented 7 years ago

@jgm,

pandoc-nightly

Regarding pandoc-nightly, I now setup a cron job to auto-deploy a daily nightly build in https://github.com/pandoc-extras/pandoc-nightly/releases. As usual, all codes are in the open and the binaries is built by Travis. I also boosted up the security a bit by using the travis cli tool to encrypt the token rather then doing it in the web interface (where the later one can potentially be attacked by collaborators doing pull request). If you don't object to it, I want to announce it in pandoc-discuss, clearly stating that it is for developers and bug reports. My hope is that it will help the issue tracker a bit — that we can point people to pandoc-nightly to download and try if their bugs is fixed in the master already. i.e. testing with the master is no longer limited to those who are able/willing to build from source. (Even for me I don't want to build from master and wait. I want to download the nightly right away to test.) By the way, this one in principle requires no maintenance — Travis does everything automatically.

pandoc-portable

Regarding pandoc-portable, not much has been changed (except boosting the security like above, changing the filename of the source code a bit). The main goal is for CI and to satisfied a need of portable packages like the OP or any one stuck with non-sudo privilege.

Other than security concern, another thing is download statistics (example of pandoc's here). But if the pandoc-portable will really be mainly used by CI, then actually this will help the download statistics: the reason is that CI downloads pandoc once per build, where one commit can trigger multiples builds at the same time, (at least for my projects and panflute, pypandoc), so that numbers goes up constantly just by CI. It might even be beneficial to tell people to setup CI pointing to pandoc-portable instead, so that we can take out the noise in download statistics, and has a better idea on the "real-user" adoption rate.

(Just some more details: CI is exactly why I later modified the script to also mirror the .deb|msi|pkg, because it will be easier for modifying the CI script at first, and might be later rewrite some other logics to take advantage of the .zip.)

Right now pandoc-portable is on hold. And when you think it is a good idea I can write a short script to get all git tags from pandoc and tell Travis to repackage all of them. (it would only takes a short amount of time from me, but Travis is going to have a busy day ;)

jgm / pandoc

[FIXED] Portable distribution #3317

pandoc-nightly

pandoc-portable

Other notes and questions