"Latest" in Bundler Asset JSON

[shawnrice]

So, I've been thinking about the way we've redone the default version in the JSON. We renamed it latest to make it work, but that means that it'll install only the latest when it is first invoked. If latest is changed in the JSON, then the older version will remain on the user's computer and act as latest.

Here's a proposal on how to change things: we change the latest value in the JSON from being a "full" asset description to being a sort of alias. The change, in the JSON would look like:

"latest":"1.0.3a",
"1.0.3a": {
...asset definination
},
etc...

Here's a possible problem: it will install the latest, still, only the first time, but, since it basically caches the path, it won't update. We could make it so that when the "update" script runs, it also checks to make sure that all the "latest" versions are actually latest.

To make it better, we could also make the latest a symbolic link to the version that is considered the latest. Then, the update script would just make an empty directory and update the symlink. The bundler would then not actually find the necessary files in the blank directory and install the new version.

To do what I've suggested we need to

1. update all the JSON files,
2. update the update.sh script to check the JSON as well,
3. double check the backend install scripts to make sure that it will behave correctly.

[deanishe]

Why does "latest" need to exist in the JSON?

Can't the bundler just grab the version with the highest version number? If a newer version is added at some point, this would be picked up during an update.

On disk, "latest" should be a symlink to the newest installed version.

[shawnrice]

Well, the only reason why it needs to be in the JSON is so that we don't have to write a function to compare version strings in every language (or maybe just in the install / update scripts).

It looks like, from what we have, the only trickier ones to parse would be with CD's 3.0.0-beta7.

So, I guess that's not that big of a deal. Good call.

But, if, if we want to push it further, then we could always allow for the bundler to accept argument commands like: 3.4.x or 3.x.

[deanishe]

I have a parser for semantic version numbers (e.g. 3.0.0-beta7). Needed it for Alfred-Workflow.

Not sure we need to go the whole hog with wildcards in version numbers.

[shawnrice]

I think I wrote one for PHP too... a while ago, but I can't remember what project it was for. Maybe I wrote it in Ruby. Or was it...? Well, fuck.

I don't think that there is too much difference when it comes to actual code to writing something that will grab the "latest" and something that would support wildcards. Without writing anything to test my thoughts, it seems like it would be only another five to ten lines of code in any relevant language. The semantic version parser is about 7/8 hog.

[shawnrice]

Oh. It was Ruby:

def parse_semantic_version(version)
    version = version.chomp
    parts   = version.split('.')
    major = parts[0]
    minor = parts[1]
    patch = parts[2][0]
    label = parts[2][2..-1]
    label = 'null' if label.nil?

    version = { :major => major, :minor => minor, :patch => patch, :label => label }
    return version
end

def compare_semantic_versions(v1, v2)
    v1 = parse_semantic_version(v1)
    v2 = parse_semantic_version(v2)

    labels = ['alpha', 'beta', 'rc', 'null']
    values = Hash[labels.map.with_index.to_a]

    return true if v1[:major] < v2[:major]
    return true if v1[:minor] < v2[:minor]
    return true if v1[:patch] < v2[:patch]
    return true if values[v1[:label]] < values[v2[:label]]
    return false
end

[deanishe]

Ruby's not my thing, but I think that one's a bit broken. Doesn't it compare everything as strings? You need to compare integer components as numbers and string components as, well, strings.

So 1.10.0 > 1.9.0 (which isn't true with string comparisons). Also 1.10.0 > 1.10.0-beta (which is also not normal sorting).

This is what I came up with (which is mostly compliant with the semantic version spec). It's a class meant to be compared directly, e.g.:

>>> Version('1.10.0') > Version('1.9.0')
True
>>> Version('1') == Version('1.0')
True
>>> Version('1.0') > Version('1.0-rc.1')
True

It's probably a bit over-the-top for the bundler's needs (e.g. no need to handle GitHub's v tag prefixes or build data).

class Version(object):
    """Mostly semantic versioning

    The main difference to proper :ref:`semantic versioning <semver>`
    is that this implementation doesn't require a minor or patch version.
    """

    #: Match version and pre-release/build information in version strings
    match_version = re.compile(r'([0-9\.]+)(.+)?').match

    def __init__(self, vstr):
        self.vstr = vstr
        self.major = 0
        self.minor = 0
        self.patch = 0
        self.suffix = ''
        self.build = ''
        self._parse(vstr)

    def _parse(self, vstr):
        if vstr.startswith('v'):
            m = self.match_version(vstr[1:])
        else:
            m = self.match_version(vstr)
        if not m:
            raise ValueError('Invalid version number: {}'.format(vstr))

        version, suffix = m.groups()
        parts = self._parse_dotted_string(version)
        self.major = parts.pop(0)
        if len(parts):
            self.minor = parts.pop(0)
        if len(parts):
            self.patch = parts.pop(0)
        if not len(parts) == 0:
            raise ValueError('Invalid version (too long) : {}'.format(vstr))

        if suffix:
            # Build info
            idx = suffix.find('+')
            if idx > -1:
                self.build = suffix[idx+1:]
                suffix = suffix[:idx]
            if suffix:
                if not suffix.startswith('-'):
                    raise ValueError(
                        'Invalid suffix : `{}`. Must start with `-`'.format(
                            suffix))
                self.suffix = suffix[1:]

        log.debug('version str `{}` -> {}'.format(vstr, repr(self)))

    def _parse_dotted_string(self, s):
        """Parse string ``s`` into list of ints and strings"""
        parsed = []
        parts = s.split('.')
        for p in parts:
            if p.isdigit():
                p = int(p)
            parsed.append(p)
        return parsed

    @property
    def tuple(self):
        """Return version number as a tuple of major, minor, patch, pre-release
        """

        return (self.major, self.minor, self.patch, self.suffix)

    def __lt__(self, other):
        if not isinstance(other, Version):
            raise ValueError('Not a Version instance: {!r}'.format(other))
        t = self.tuple[:3]
        o = other.tuple[:3]
        if t < o:
            return True
        if t == o:  # We need to compare suffixes
            if self.suffix and not other.suffix:
                return True
            if other.suffix and not self.suffix:
                return False
            return (self._parse_dotted_string(self.suffix) <
                    self._parse_dotted_string(other.suffix))
        # t > o
        return False

    def __eq__(self, other):
        if not isinstance(other, Version):
            raise ValueError('Not a Version instance: {!r}'.format(other))
        return self.tuple == other.tuple

    def __ne__(self, other):
        return not self.__eq__(other)

    def __gt__(self, other):
        if not isinstance(other, Version):
            raise ValueError('Not a Version instance: {!r}'.format(other))
        return other.__lt__(self)

    def __le__(self, other):
        if not isinstance(other, Version):
            raise ValueError('Not a Version instance: {!r}'.format(other))
        return not other.__lt__(self)

    def __ge__(self, other):
        return not self.__lt__(other)

    def __str__(self):
        vstr = '{}.{}.{}'.format(self.major, self.minor, self.patch)
        if self.suffix:
            vstr += '-{}'.format(self.suffix)
        if self.build:
            vstr += '+{}'.format(self.build)
        return vstr

    def __repr__(self):
        return "Version('{}')".format(str(self))

[shawnrice]

Ruby's not my thing, but I think that one's a bit broken. Doesn't it compare everything as strings? You need to compare integer components as numbers and string components as, well, strings.

Well, it is better to do what you say, but, if I remember correctly, it worked because strings can still be compared with a greater than, etc... method. Granted, I hadn't put it through any rigorous testing, and the thing that I wrote it for hasn't been put into use at all yet. I'll definitely do more testing on it before it becomes part of the bundler.

[deanishe]

You can compare strings with >, <, etc., but it's an alphabetical comparison:

irb(main):003:0> "9" > "10"
=> true
irb(main):004:0> 9 > 10
=> false

[shawnrice]

Touché.

Again, from memory, all my tests were integers below 9. I'll make sure that the code gets an overhaul.

[deanishe]

Yeah, it's a bit of a pig. But I think the above algo works (I used relatively comprehensive tests).

Regarding wildcards (if we really have to), I think it might be a good idea to convert, say, v = 1.4.x to v >= 1.4 and v < 1.5 or v = 1.x to v >= 1.0 and v < 2.

[shawnrice]

I'll copy the algorithim in code translations as closely as possible (and, of course, write real tests).

Re: wildcards: I could go either way for x or the syntax above. I know that composer uses the x, but most everything Python and Ruby use what you indicate above, which is much more powerful, but I'm assuming that it's also much harder to write (i.e. 1.5 hogs, to use the same unit of measure).

[deanishe]

What's "composer"? Is that like gem/pip for PHP?

I don't think pip supports that kind of specification. You can do v >= 1.1.1 or v == 1.1.1, but I don't think you can also specify a maximum version (or wildcards).

I didn't explain myself very well. What I meant was, if we go with the syntax v = 1.1.x, we should treat that internally as v >= 1.1 and v < 1.2 in order to determine which version to install.

Or we could convert x to .+ and . to \., and run a regex match across available versions, then choose the highest matching version.

In fact, that might be a better idea. It would fail on any version strings that contain 01 instead of 1, however, i.e. it wouldn't be a "proper" version comparison.

[shawnrice]

Yep. Composer is gem/pip for PHP. It's pretty rad, actually. You even just have a composer.json file in a directory that functions just like a gemspec or a requirements.txt. Then, you type composer install, and it grabs everything for you and locks it. You can then update with subsequent commands. Packagist is the same as Rubygems or PyPi.

Oh, yes, you didn't explain yourself well. With your clarification: yes. That is what we will do.

I'm not positive I follow you on the regex, especially because I'm trying to think of how PCRE functionality is coded into Bash, PHP, and Ruby. Why can't we just split it on the dots and compare? It seems easier.