Hoogle server returns the url to the oldest package (ranked by pvp) when multiple package versions live on the same server

[fredericcogny]

Hi Neil

Hope this finds you well.

Background We've set-up our own doc server with hoogle for the packages we develop internally One of our requirements was too keep older versions of the docs on the server as well. So we now have side by side folders like: mylib-1.0.0, mylib-1.0.1, mylib-2.0.0 etc ... So when we run hoogle generate they all get captured in the db and that's all good.

Issue The issue we have though comes from the fact that when we get back the results of the search, the URL that is displayed points to the oldest package. The desired behaviour would be to have it pointing to the latest instead. [the whole list of modules and packages appear fine just below]

Solution? Looking at the module Action.Server I think this is coming from the function showResults (pasted below) where, for each result, you pattern match against the head of the list of Targets. (cf l 166). So naturally the targetURL is the first of your Targets. If we were to order that list by a "reverse"-natural version policy ordering (or even simpler maybe, by reverse lexicographic ordering) it would take care of our problem. If you wish I'm happy to give it a try but first I wanted to check with you whether the [[Targets]] that come to showResults are pre-ordered following some other rules. In which case, can those rules be modified or was the choice of the order deliberate for some other reason ?

Thanks in advance

Fred

showResults :: Bool -> Maybe FilePath -> [(String, String)] -> [Query] -> [[Target]] -> String
showResults local haddock args query results = unlines $
    ["<h1>" ++ renderQuery query ++ "</h1>"
    ,"<ul id=left>"
    ,"<li><b>Packages</b></li>"] ++
    [tag_ "li" $ f cat val | (cat,val) <- itemCategories $ concat results, QueryScope True cat val `notElem` query] ++
    ["</ul>"] ++
    ["<p>No results found</p>" | null results] ++
    ["<div class=result>" ++
     "<div class=ans><a href=\"" ++ showURL local haddock targetURL ++ "\">" ++ displayItem query targetItem ++ "</a></div>" ++
     "<div class=from>" ++ showFroms local haddock is  ++ "</div>" ++
     "<div class=\"doc newline shut\">" ++ targetDocs ++ "</div>" ++
     "</div>"
    | is@(Target{..}:_) <- results]

[fredericcogny]

I've tested locally: a simple change on line in Action.Serverto: | is@(Target{..}:_) <- map reverse results] returns the desired result

[ndmitchell]

Hi Fred - all good here. Hope you and your family are well too.

For generating side by side packages with different versions, how are you doing that? Do they have a @package mylib-1.0.0 in the text file? Or do they all have the same @package name?

The targets are indeed sorted, and they are sorted by https://github.com/ndmitchell/hoogle/blob/master/src/Input/Reorder.hs. My guess is that I should tweak:

    concatMap snd $ sortOn ((packageOrder &&& id) . fst) $ map rebase $ splitIPackage xs

To become something like:

    concatMap snd $ sortOn ((packageOrder &&& packageNameWithoutversion &&& reverseByPackageVersion) . fst) $ map rebase $ splitIPackage xs

But I honestly hadn't considered multiple versions of one library, and am actually having difficulty persuading Hoogle to support it - so let me know your trick and I'll confirm that idea works.

[fredericcogny]

Hoogle does not seem to need any persuasion and seems quite fine with it on my end :) Also there is no trick per se. We just:

• generate the docs of your package (with dependencies) using stack haddock
• tar them and upload them onto our doc server
• server un-tars them and places the packages in the same folder as all the other packages

And it works...

I've reproduced the same behaviour locally by doing the following on two different package versions

docDir="$HOME/git/Haskell/doc/"
stack haddock
tar -czvf "$docDir/doc.tgz" -C "$(stack path --local-doc-root)" --exclude "./all" .
tar -xkf "$docDir/doc.tgz" -C "$docDir"
rm "$docDir/doc.tgz"
hoogle generate --local="$docDir"
sudo hoogle server --local

To answer your question: After doing that they all have the same @packagename in the text files. For instance on the package safecopy (that was in my dependencies) I have

in the folder safecopy-0.9.2

@package safecopy
@version 0.9.2

in the folder safecopy-0.9.3.1

@package safecopy
@version 0.9.3.1

then if you search for safeGetfor instance: (http://127.0.0.1/?hoogle=safeGet&scope=set%3Aall) you get a link to 0.9.2:

<a href="file/Users/Fred/git/Haskell/doc/safecopy-0.9.2/Data-SafeCopy-Internal.html#v:safeGet"><span class="name"><b>safeGet</b></span> :: SafeCopy a =&gt; Get a</a>

and right below the links to the packages and the modules appear each twice in order of the version

[ndmitchell]

I finally figured out why I wasn't seeing what you were - my Chrome was caching the output from before I copy/pasted one package, so I was seeing a view of a single package. After spawning the server on a different port I see what you are seeing.

The problem is that Hoogle doesn't keep track of package version or use it anywhere. It tolerates multiple packages (which is actually a bit of a surprise). The reason you always get 1.0 before 2.0 is because I use listDirectoryRecursive to find the initial packages, and that usually returns sorted results, and then 1.0 stays before 2.0 forever. As a quick hack I've done a reverse on that list, which will be pretty close to a fix. Can you check that does what you were hoping?

Next refinement is to sort based on version numbers, so 2.10 > 2.9.

[fredericcogny]

Thanks. That worked locally for packages with numbers up to 9. However just like you mentioned in your last comment I see 1.2.8 appear before 1.2.16. Not sure however how easy that is for you, so not a big deal if this is too much work.

[ndmitchell]

I've fixed the 1.2.8 vs 1.2.16 thing. Can you check it works for you and I'll make a release. I've also added a test case so it should continue to work in the future.

[fredericcogny]

Hi Neil Happy Easter I just checked and it works fine for me now. Thanks again

[ndmitchell]

Happy Easter to you too. Glad it worked - I've released hoogle-5.0.10 with the fix included.