Show Dash user contrib docsets in Zeal

[jboadas]

Just trying to see the user contrib docsets but can't see it on the list

[sirajulm]

Which docset creates the problem? Have you checked with some other user contributed docsets? Are the config files proper as specified here: Generating Dash Docsets

[josephfinlayson]

You do it this way:

https://twitter.com/J4lf/status/553489605958594560

[wosc]

Thanks @josephfinlayson for pointing out that "feed" can also be a Tarball!

However, most Tarballs are removed from the User Contributions repository and uploaded to the @Kapeli CDN, which is not publicly accessible AFAICT, so what you need to do is trawl through the Pull Requests and look for the Tarballs there.

This is a bit of annoying manual work, but I don't know what a more convenient solution could look like, since I don't think it would be fair to ask @Kapeli to offer these docsets for us non-paying Zeal users via bandwidth/resources that @Kapeli is paying for.

[Kapeli]

Sorry for not seeing this issue earlier!

Actually I'd be glad if Zeal would use these docsets as well. I already allow access to the other (main) docsets, so I don't see a reason why not to do the same with the user contributed ones.

You can get a json of all the available user contributed docsets at http://sanfrancisco.kapeli.com/feeds/zzz/user_contributed/build/index.json

Once you have that, you can get to the docset archives like this:

http://<mirror>.kapeli.com/feeds/zzz/user_contributed/build/<key name inside json>/<archive name>

Examples:

http://newyork.kapeli.com/feeds/zzz/user_contributed/build/Atom/Atom.tgz
http://london.kapeli.com/feeds/zzz/user_contributed/build/SDL/org.libsdl.sdl20.docset.tgz

The icons in the json are base64 encoded.

Regarding the <mirror> part, you're supposed to choose the closest/best mirror. I've given details about how to do that to @trollixx.

[trollixx]

Thank you, @Kapeli!

I am working on a revamped docset manager for Zeal, which will include user contributed docsets.

[lowjoel]

Would RDoc/Yard documentation support (like Dash) belong to this ticket also?

[dkniffin]

:+1: to rdoc support in zeal.

[NoAnyLove]

Since I was looking for docset for matplotlib, it's very happy to find this and finally soveld my problem. Thank you @Kapeli !

[lowjoel]

@Kapeli The instructions on the Dash site only show how to get docsets for published gems Yardoc/Rdoc, but doesn't document how to generate them. Any way to document that in the meanwhile?

[Kapeli]

Nope, sorry. The tool Dash uses to generate docs from Ruby Gems is not really usable if it's not run by Dash. I'll open-source it and package it into something everyone can use if more users ask.

The tool is written in Objective-C, so it won't be usable outside of OS X even if I release it.

[lowjoel]

@Kapeli I'm looking forward to it, if it's released. Even if it's in Obj-C I think people would write ports.

In the absence of that, does your docset license allow Zeal to use the Gem documentation from your site?

[Kapeli]

In the absence of that, does your docset license allow Zeal to use the Gem documentation from your site?

No idea what you mean. What gem documentation?

[lowjoel]

@Kapeli Dash allows users to download Yardoc/Rdoc documentation for gems, running through your generator on your end. Since your docsets are licensed out for Zeal, does that allow Zeal to request for Yardoc/Rdoc docsets through your service too?

(Maybe I'm missing something)

[Kapeli]

The Ruby Gem docsets are not generated on my servers. Dash (i.e. the client) downloads the Ruby Gem from rubygems.org, then generates the documentation using yard and then generates the docset. This all happens locally. My servers are not involved.

[lowjoel]

I see, so then for that we would need that tool to generate the proper docsets... Let's see what demand is like.

I was really looking forward to that since I'm doing Rails apps quite a lot nowadays...

[trollixx]

I've created #285 to track progress on Ruby documentation generation. This issue is about supporting user-contributed docsets from Dash.

[prusswan]

It would be "good to have"

[kidd]

Trying to implement the ability to get custom docsets, I find that by building the string as explained in this thread, I get things like "http://newyork.kapeli.com/feeds/zzz/user_contributed/build/Google App Engine - Python/GAE-Python.tgz" which have spaces, and they don't seem to exist. tried urlencoding and didn't work either... @Kapeli , any hints?

[trollixx]

It should be http://newyork.kapeli.com/feeds/zzz/user_contributed/build/Google_App_Engine-Python/GAE-Python.tgz.

[Kapeli]

You're supposed to add the "key" of the dictionary entry (first line in the sample below), not the value under name. That would be Google_App_Engine-Python instead of Google App Engine - Python.

    "Google_App_Engine-Python" : {     // <--- use this
      "author" : {
        "name" : "TZ Martin",
        "link" : "https:\/\/twitter.com\/tzmartin"
      },
      "aliases" : [
        "Google App Engine Python",
        "AppEngine",
        "Google Cloud"
      ],
      "archive" : "GAE-Python.tgz",
      "specific_versions" : [
        {
          "version" : "1.6.9.445",
          "archive" : "versions\/1.6.9.445\/GAE-Python.tgz"
        }
      ],
      "version" : "1.6.9.445",
      "name" : "Google App Engine - Python"
    },

[despairblue]

@Kapeli do the user contrib docs also have xml feeds?

At the moment if one wants to use one of the user contrib docs one has to download the archive and extract it manually into zeals docset folder, since zeal's add feed option only accepts xml feads not the tgz links directly.

[Kapeli]

@Kapeli do the user contrib docs also have xml feeds?

No, sorry.

[despairblue]

Ok, but still, thanks for making them available anyhow :smiley:

[kidd]

I've been dabbling with this issue, and created a little service (hosted in heroku) to retrieve a list of all the contrib docsets in a straight format (parsing xml and json is not the funniest thing in emacs lisp). It's still unfinished, and subject to changes, and in fact, I'd like to make it retrieve also the official ones.

https://dashes-to-dashes.herokuapp.com/docsets/contrib

@Kapeli, Is it ok to follow this path? or you prefer we don't re-distribute the list in any other form appart from the official one? (completely understandable due to adding an extra part in the game is not always the best solution and changes in the api will have to be synched by both).

[despairblue]

@kidd cool stuff, but I'm lacking some background here as I don't see the purpose. This is still json and not parsable by zeal. Or am I mistaken?

[kidd]

I was doing it for https://github.com/areina/helm-dash, and my idea was to end with only json, and directly parseable like [{"name": "foo", "archive": "http://lala.com/foo.tgz"}...] . My plans were to create this intermediate layer for both official and user-created docsets.

As I can't create the xml feeds myself... So it's an alternative, and if we all agree on one way of showing this info, I can adapt both helm-dash and dashes-to-dashes.

[trollixx]

Wow, that's the most commented issue so far. :)

I do have plans to add the support for the community contributed docsets in the nearest time. Importing the list to our database is a really simple task. We already have a working process for official docsets, so Zeal users see updates almost immediately as they get released for Dash.

The problem is our docset management UI. It's primitive and adding another hundred of docsets to it would make it hard to use. So, Zeal definitely needs a better docset manager, and that will take some time to implement.

I think I'll add importing of user docsets on the server-side some time this or next week, but no any estimates on that landing as a stable release yet.

[Kapeli]

@kidd It's ok with me. Feel free to redistribute and reformat the json in any way you want for helm-dash.

[kidd]

Hi. the PR finally landed, and in https://github.com/kidd/dashes-to-dashes there's the code for this reformatting of the docsets info. For now we're only using it for user-contribs ( https://dashes-to-dashes.herokuapp.com/docsets/contrib ) but probably we'll use it also for the official ones.

Just FYI

[crmne]

I created a little program in Python 3 and easygui (pip3 install easygui) to download and extract the user docsets in Zeal's docsets folder. Hope it helps!

https://gist.github.com/crmne/3fe84c05013fa87d74a8

[abn]

@crmne :+1:

[DerekTurtleRoe]

:+1: I hope user contributed Dash docsets are added, as this means I don't need to keep two docs apps on my PC currently. If Zeal gets a dark theme and user contributed docs, I will switch over entirely. :smile:

[ssokolow]

@vgturtle127 In the mean time, you can use crmne's script for loading user-contributed docsets into Zeal: https://gist.github.com/crmne/3fe84c05013fa87d74a8

I can confirm that it works. (It's not as comfortable as what I'd have done, but crmne actually had time to make something.)

[DerekTurtleRoe]

@ssokolow I can't seem to get that to work on my system, but that is probably due to my current system setup. When I re-install Windows I will try it again.

[ssokolow]

@vgturtle127 Hmm. It's likely the author didn't test on Windows and I don't have anything newer than Windows XP to test on, after a quick skim over the code, I can say that any incompatibility should be fairly trivial.

What does it say when it doesn't work?

[DerekTurtleRoe]

@ssokolow Well, when I run it it just doesn't do anything at all. :laughing:

I tried running it as admin as well, but that also didn't do anything.

I tried to go through and see if there were any Windows compatibility issues as well, but I can't see anything that would cause it to just not run at all...

It may be my wonky setup with MSYS, Cygwin, MinGW, etc. and also my computer has had multiple versions of Python installed (including development versions), so that could be an issue as well. I suppose I could try uninstalling all those and re-installing the latest versions, but I fear the issue may be in the registry or in AppData/ProgramData. Hmm....

[ssokolow]

Try running it in a cmd.exe window first. It's probably dying with an error message that you don't have a chance to read.

[DerekTurtleRoe]

@ssokolow I tried that too, it does the same thing.

Can't wait for the Windows 10 Anniversary update so I can get Bash in the command-line. :wink:

[ssokolow]

Hmm. Let's make a little checklist:

1. Does the path produced by python -c "import os; print(os.path.expanduser('~/.config/Zeal/Zeal.conf'))" exist or is it failing because it's looking for a Linux-only path?
2. Confirm it's being run in Python 3.x. I know that's mandatory from an oversight of my own.
3. Is EasyGUI installed in Python 3.x?

[DerekTurtleRoe]

@ssokolow

1. I do have a Zeal.conf file, assuming that it created it in the expected directory.
2. It is, and I tried it in 2.x for lolz and it didn't work...
3. Yes, and I tried re-installing that as well.

I think my Python installation is bonked, let me try uninstalling some stuff and cleaning up a little...

[zsoltika]

Some of You might be interested in this https://github.com/zsoltika/feeds . It's just a basic workaround for the above solutions/tips:

1. forking @Kapeli 's feeds repo
2. writing a lame sed/awk parser around his JSON to generate the XMLs

So in this repo the actual docsets are available for Zeal via XML feeds.

[despairblue]

@zsoltika This is awesome! Is this kept up-to-date automatically (by travis or similar) or periodically by you?

[zsoltika]

@despairblue as far as I see, bash is not yet supported by travis, and the problem is, that after generating the new XMLs, one would have to push to github too. Anyhow I'm looking into another solution (like creating and updating a public gist, or something similar).

[ssokolow]

@zsoltika

Bash is a de facto standard component of Linux and too many scripts depend on it for them to exclude it from the Ubuntu images they use for building. Even if that weren't the case, you're allowed to install things (eg. via apt-get) as part of setting up to run your tests.

Just make sure you're choosing a profile that uses Linux rather than OSX to build and you're good.

(Aside from choosing Linux vs. OSX, the predefined profiles are just for efficiency. They just determine which packages come pre-installed and which build settings are default if you don't specify them. People were using languages like Rust on Travis long before they gained official support via apt-get and other download mechanisms.)

People also use Travis to build things and push them elsewhere all the time. They've got an entire section of the documentation (Deployment) dedicated to it.

Here's the part on deploying via git push.

...and here's a StackOverflow thread which explains a couple of different ways to deal with securely giving Travis-CI permission to push to a GitHub repo (in their example, a GitHub Pages site):

https://stackoverflow.com/questions/23277391/how-to-publish-to-github-pages-from-travis-ci

[zsoltika]

Thanks for the info, I'm looking into this.

On Tue, Jun 14, 2016 at 3:05 PM, Stephan Sokolow notifications@github.com wrote:

@zsoltika https://github.com/zsoltika

Bash is a de facto standard component of Linux amd too many scripts depend on it for them to exclude it from the Ubuntu images they use for building. Even if that weren't the case, you're allowed to install things (eg. via apt-get) as part of setting up to run your tests.

(The predefined profiles are for efficiency. People were using languages like Rust on Travis long before they gained official support via apt-get and other download mechanisms.)

People also use Travis to build things and push them elsewhere all the time. They've got an entire section of the documentation (Deployment) dedicated to it.

Here's the part on deploying via git push https://docs.travis-ci.com/user/deployment/custom/.

...and here's a StackOverflow thread which explains a couple of different ways to deal with securely giving Travis-CI permission to push to a GitHub repo (in their example, a GitHub Pages site):

https://stackoverflow.com/questions/23277391/how-to-publish-to-github-pages-from-travis-ci

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/zealdocs/zeal/issues/170#issuecomment-225874678, or mute the thread https://github.com/notifications/unsubscribe/AAFeR2HntbN6G6srto2PQEYRbOwgoIIwks5qLqcrgaJpZM4CI8XR .

[sciunto]

A quick comment. Dash can't fix wrong naming (and they obviously don't want to fix the bug). It results that at least one xml file has a wrong name in https://github.com/zsoltika/feeds

See https://github.com/Kapeli/Dash-User-Contributions/pull/984

[thuandt]

Does Zeal 0.3.0 support Dash Contrib Docsets?

I saw github tag but seem src code still isn't push to launchpad (PPA)

[trollixx]

Not yet, otherwise this ticket would be closed. I'll be updating packages over a few days.

[xantiagoma]

This may help someone, Based on https://kapeli.com/feeds/zzz/user_contributed/build/index.json

I've created this tool to get feed URLs to add to Zeal https://zealusercontributions.now.sh/ For example, with Emacs the feed URL is https://zealusercontributions.now.sh/api/docsets/Emacs.xml Also in Home page you can get the icons to download them into the docset folder and all avaliable docsets and its URLs.

If someone wants to create its own page or use as a base the code is public: https://github.com/xantiagoma/zealusercontributions

Mirror: http://zealusercontributions.now.sh/

[trollixx]

@sntiagoma this is a pretty cool workaround for my laziness :) Thanks for sharing!