SHA - Unable to confirm integrity of downloaded file

[sathomps]

I've updated my version of Jenkins to 2.131, using the juseppe-plugin and the update-sites-manager-plugin. When trying to install an internal plugin hosted and produced by the plugins above, I get the following error in Jenkins.

Attempt to verify a downloaded file (xxx.jpi.tmp) using SHA-512 failed since your configured update site does not provide this checksum. Falling back to weaker algorithms.
Jul 10, 2018 1:29:38 PM SEVERE hudson.model.UpdateCenter$DownloadJob run
Failed to install XXX
java.io.IOException: Unable to confirm integrity of downloaded file, refusing installation

[pcooke2002]

I am having this problem... is anyone else experiencing this? Do we have a fix? looks like they updated jenkins to require more stringent hash check

[sathomps]

The fix is actually pretty simple, I haven't had a chance to submit a pull request.

plugin.json

{
    "type": "object",
    "properties": {
        "excerpt": {
            "type": "string"
        },
        "url": {
            "type": "string"
        },

        "releaseTimestamp": {
            "type": "string"
        },

        "buildDate": {
            "type": "string"
        },

        "version": {
            "type": "string"
        },

        "title": {
            "type": "string"
        },

        "wiki": {
            "type": "string"
        },

        "compatibleSinceVersion": {
            "type": "string"
        },

        "requiredCore": {
            "type": "string"
        },

        "builtBy": {
            "type": "string"
        },

        "name": {
            "type": "string"
        },

        "group": {
            "type": "string"
        },

        "sha1": {
            "type": "string"
        },

        "sha512": {
            "type": "string"
        },

        "dependencies": {
            "type": "array",
            "items": {
                "$ref": "dependency.json"
            }
        },

        "developers": {
            "type": "array",
            "items": {
                "$ref": "developer.json"
            }
        }

    }
}

HPI.java

/**
 * HPI Class for extracting HPI information from hpi file.
 *
 * @author JunHo Yoon
 */
public class HPI {
    private static final Logger LOG = LoggerFactory.getLogger(HPI.class);
    private static final Pattern DEVELOPERS_PATTERN = Pattern.compile("([^:]*):([^:]*):([^,]*),?");

    private static final String OPTIONAL_DEPENDENCY = ";resolution:=optional";

    public static final String PLUGIN_VERSION = "Plugin-Version";
    public static final String PLUGIN_WIKI_URL = "Url";
    public static final String PLUGIN_TITLE = "Long-Name";
    public static final String PLUGIN_NAME = "Short-Name";
    public static final String PLUGIN_COMPATIBLE_SINCE_VERSION = "Compatible-Since-Version";
    public static final String PLUGIN_REQUIRED_JENKINS_VERSION = "Hudson-Version";
    public static final String PLUGIN_BUILT_BY = "Built-By";
    public static final String PLUGIN_DEPENDENCIES = "Plugin-Dependencies";
    public static final String PLUGIN_DEVELOPERS = "Plugin-Developers";
    public static final String GROUP_ID = "Group-Id";

    public static final String MANIFEST_PATH = "META-INF/MANIFEST.MF";

    public static Plugin loadHPI(final File file) throws IOException {
        final JarFile jarFile = new JarFile(file);
        final long timestamp = jarFile.getEntry(MANIFEST_PATH).getTime();

        final Plugin hpi = HPI.from(file, jarFile.getManifest().getMainAttributes(), timestamp);

        final String wiki = getWiki(file);
        if (StringUtils.isNotBlank(wiki)) {
            hpi.setWiki(wiki);
        }

        jarFile.close();

        return hpi.withExcerpt(getExcerpt(file));
    }

    private static Plugin from(final File file, final Attributes attributes, final long timestamp) throws IOException {
        final Digests digests = digests(file);

        return new Plugin()
                        .withReleaseTimestamp(releaseTimestampDateFormat().format(timestamp))
                        .withBuildDate(buildDateTimeFormat().format(timestamp))
                        .withName(
                                        notBlank(attributes.getValue(PLUGIN_NAME), "Plugin short name can't be empty"))
                        .withVersion(
                                        defaultIfBlank(
                                                        attributes.getValue(PLUGIN_VERSION),
                                                        attributes.getValue(Attributes.Name.IMPLEMENTATION_VERSION)))
                        .withGroup(attributes.getValue(GROUP_ID))
                        .withWiki(attributes.getValue(PLUGIN_WIKI_URL))
                        .withTitle(attributes.getValue(PLUGIN_TITLE))
                        .withCompatibleSinceVersion(attributes.getValue(PLUGIN_COMPATIBLE_SINCE_VERSION))
                        .withRequiredCore(attributes.getValue(PLUGIN_REQUIRED_JENKINS_VERSION))
                        .withBuiltBy(attributes.getValue(PLUGIN_BUILT_BY))
                        .withDependencies(getDependencies(attributes))
                        .withDevelopers(getDevelopers(attributes))
                        .withSha1(digests.sha1)
                        .withSha512(digests.sha512);
    }

    private static Digests digests(final File file) {
        final Digests digests = new Digests();
        try (FileInputStream fin = new FileInputStream(file)) {
            final MessageDigest sha1 = DigestUtils.getSha1Digest();
            final MessageDigest sha512 = DigestUtils.getSha512Digest();
            final byte[] buf = new byte[2048];
            int len;
            while ((len = fin.read(buf, 0, buf.length)) >= 0) {
                sha1.update(buf, 0, len);
                sha512.update(buf, 0, len);
            }

            digests.sha1 = new String(Base64.encodeBase64(sha1.digest()), "UTF-8");
            digests.sha512 = new String(Base64.encodeBase64(sha512.digest()), "UTF-8");
        } catch (final Exception ex) {
            LOG.error(ex.getMessage(), ex);
        }
        return digests;
    }

    private static SimpleDateFormat releaseTimestampDateFormat() {
        final SimpleDateFormat releaseFormat = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss'.00Z'", Locale.US);
        releaseFormat.setTimeZone(TimeZone.getTimeZone("UTC"));
        return releaseFormat;
    }

    private static SimpleDateFormat buildDateTimeFormat() {
        return new SimpleDateFormat("MMM dd, yyyy", Locale.US);
    }

    protected static List<Dependency> getDependencies(final Attributes attributes) throws IOException {
        final String deps = attributes.getValue(PLUGIN_DEPENDENCIES);
        if (deps == null) {
            return Collections.emptyList();
        }

        final List<Dependency> result = new ArrayList<>();
        for (final String token : deps.split(",")) {
            result.add(parseDependencyFrom(token));
        }
        return result;
    }

    protected static Dependency parseDependencyFrom(String token) {
        final boolean optional = token.endsWith(OPTIONAL_DEPENDENCY);
        if (optional) {
            token = substringBefore(token, OPTIONAL_DEPENDENCY);
        }

        final String[] pieces = token.split(":");

        return new Dependency()
                        .withName(pieces[0])
                        .withVersion(pieces[1])
                        .withOptional(optional);
    }

    protected static List<Developer> getDevelopers(final Attributes attributes) throws IOException {
        final String devs = attributes.getValue(PLUGIN_DEVELOPERS);
        if (isBlank(devs)) {
            return Collections.emptyList();
        }

        final List<Developer> result = new ArrayList<>();
        final Matcher matcher = DEVELOPERS_PATTERN.matcher(devs);
        int totalMatched = 0;
        while (matcher.find()) {
            result.add(new Developer()
                            .withName(trimToEmpty(matcher.group(1)))
                            .withDeveloperId(trimToEmpty(matcher.group(2)))
                            .withEmail(trimToEmpty(matcher.group(3))));
            totalMatched += matcher.end() - matcher.start();
        }
        if (totalMatched < devs.length()) {
            // ignore and move on
            LOG.error("Unparsable developer info: '{}'", devs.substring(totalMatched));
        }
        return result;
    }

    protected static String getWiki(final File hpiFile) {
        try {
            final String baseName = FilenameUtils.getBaseName(hpiFile.getName());
            final File exceptFile = new File(hpiFile.getParent(), baseName + ".wiki");
            if (exceptFile.exists()) {
                return FileUtils.readFileToString(exceptFile, "UTF-8");
            } else {
                return "";
            }
        } catch (final Exception e) {
            return "";
        }
    }

    protected static String getExcerpt(final File hpiFile) {
        try {
            final String baseName = FilenameUtils.getBaseName(hpiFile.getName());
            final File exceptFile = new File(hpiFile.getParent(), baseName + ".info");
            if (exceptFile.exists()) {
                return FileUtils.readFileToString(exceptFile, "UTF-8").replace("\r\n", "<br/>\n");
            } else {
                return "";
            }
        } catch (final Exception e) {
            return "";
        }
    }

    private static class Digests {
        private String sha1;
        private String sha512;
    }

}

[pcooke2002]

Hi thanks for posting this. My employer prohibits me from modifying any OSS tools or contributing to any OSS source code.

[grazius]

hi all this fix need to be integrated or juseppe will stay unusable with last version of jenkins.

[sathomps]

As it appears that this project is no longer being maintained, I'll create a fork this weekend with the update.

[lanwen]

yes, sorry, have no time to maintain it anymore, as don't use it by myself for a long time

[lanwen]

@sathomps actually you can send PR if you find a solution - will merge it

[sathomps]

Thanks @lanwen for the update, understandable with the time commitments.

As I need this tool at work, I'll take over the project if you don't mind?

[lanwen]

sure, I can give you write permissions

[sathomps]

Thanks!

[lanwen]

@sathomps done, but please always submit changes using PRs

[grazius]

@sathomps hello, now you can add this fix ? you can generate the docker hub image also ?

[lanwen]

docker hub image build automated from master and tags

[sathomps]

@grazius I will be incorporating this change on Sunday afternoon.

[sathomps]

Please reopen if you continue to have issues.

[grazius]

Nice thx Can you also update the docker hub image with this fix?

[lanwen]

@sathomps test failed for some reason https://hub.docker.com/r/lanwen/juseppe/builds/bntmzmpx2tj6vxqml2jhgtu/

[sathomps]

Hmm - the tests on the merge said it was successful.

The logs referenced to the Docker build failed due to a timeout. How do you trigger a new docker build, not familiar with the process.

[lanwen]

You can turn off tests on package phase in pom

[sathomps]

Sure I know how to turn off tests in the pom but it's not really a solution :)

I'd like to trigger another travis build but it appears I don't have owner access to this repository - can't view settings.

[lanwen]

build is automated on docker hub side

[lanwen]

since issue exists only while docker build, i suggest to turn the tests off while creating the container - as it adds no value after PR and master are tested

[sathomps]

Docker build has been updated https://hub.docker.com/r/lanwen/juseppe/builds/bgtw8cadyb6vn2dzdcpjudi/

[pcooke2002]

Hi when will there be a release that includes this code? I see it checked in some time back and no new release???

[sathomps]

Code is there and has been merged.

[caohhung]

Hi, latest release doesn't contain this fix https://github.com/jenkinsci/juseppe/releases

cc @lanwen

[vihoti]

Hi @sathomps I am using v1.2.2 and for versions of Jenkins LTS >= 2.319.1 we are getting the same error message. Should I open a new issue or does it suffice to mention it here?