SitemapLoader fails to load a CSV without header line

[kazurayam]

v 0.9.5 https://github.com/kazurayam/inspectus/blob/0.9.5/src/main/java/com/kazurayam/inspectus/materialize/discovery/SitemapLoader.java

Given a CSV file, as

/index.html,By.xpath: //img[@alt='umineko']
/repositories.html,By.xpath: //footer
/proverbs.html,By.xpath: //footer

(please note that this file does NOT have a header line)

then the following code

rivate List<Target> getTargetList(URL baseTopPageURL, Path targetFile)
            throws InspectusException {
        assert Files.exists(targetFile) : targetFile + " is not found";
        Target baseTopPage = Target.builder(baseTopPageURL).build();
        SitemapLoader loader = new SitemapLoader(baseTopPage);
        // loader.setWithHeaderRecord(false);
        Sitemap sitemap = loader.parseCSV(targetFile);
        System.out.println("sitemap=" + sitemap);
        return sitemap.getBaseTargetList();
    }

The returned List is emtpy.

This is because it does not call loader.setWithHeaderReport(false).

This is fragile; difficult to debug; how can we remember that the line is such significant.

[kazurayam]

LINE#83-100:

            try {
                Iterable<CSVRecord> records = csvFormat.parse(in);
                for (CSVRecord record : records) {
                    Map<String, String> map = record.toMap();
                    if (map.get("url") != null && map.get("url").length() > 0) {
                        URL url = resolveUrl(map.get("url"));
                        Handle handle = Handle.deserialize(map.get("handle"));
                        map.remove("url");
                        map.remove("handle");
                        Target t =
                                Target.builder(url).handle(handle)
                                        .putAll(map).build();
                        sitemap.add(t);
                    }
                }
            } catch (IOException e) {
                throw new InspectusException(e);
            }

This code gives no warning message even if it found 0 Target to return.

It should give some sensible warning message and remind us that we may need to call loader.setWithHeaderReport(false).

[kazurayam]

done at v0.9.6