GitHub Pod Parsing

[zoffixznet]

I'm pretty new to the community, so I don't know if this was brought up in the past....

GitHub doesn't parse P6 pod and shows errors when viewing docs in this repo. I contacted GitHub and this was their response:

Hi ZZ,

It's not possible to disable it, but the implementation we use is open source1. We'd love to discuss ideas for supporting .pod in both Perl 5 and Perl 6. Is there a way to detect if a pod file is targeting a specific version?

So if we could come up with some way to disambiguate between P5 and P6 pods, we could fix the issues in the repo (as well as any potential P6 modules that use P6 pod).

[azawawi]

:+1:. I was asking about it earlier in #perl6. Here is the file that does perl5 pod processing in github:

https://github.com/github/markup/blob/master/lib/github/markups.rb#L41

[MadcapJake]

As @Skarsnik mentioned on irc, why not use the .pod6 extension? Or couldn't you just require a v6 pragma? Lastly, there's always gitattributes!

[zoffixznet]

:+1: on using .pod6

[AlexDaniel]

So?

[AlexDaniel]

OK, now what?

[awwaiid]

Now do a PR against https://github.com/github/markup/blob/master/lib/github/markups.rb#L41 to differentiate and render pod6

[zoffixznet]

https://github.com/github/markup/issues/907

[AlexDaniel]

Unassigning @coke because it seems like he agreed to work on file renaming but not the github markup thing. Assign yourself again if that's not the case.

[zoffixznet]

GitHub replied about setting up Perl 6 Pod renderer: https://github.com/github/markup/issues/907#issuecomment-246855769

[coke]

Per the github/markup ticket, they could use someone from our team to make a PR:

You're certainly more than welcome to perform a PR with tests, and update the .travis.yml file so that our CI test suite verifies that Perl6 documentation is being generated.

[zoffixznet]

Yeah, but they'd need to install Perl 6 for any of that to be of use. They talk about Ruby glue, but how do we get the stuff to glue with? And as far as my understanding goes: it'd be (a) a fully functional Perl 6 compiler that would compile arbitrary code from strangers; and (b) they don't pre-build the pages, right? So how would they cope with (a) and having potentially thousands of pages rendered with slow and resource hungry compiler?

Do we have any option of limiting the pod parser to just pod and not arbitrary code?

[toolforger]

Wait... what... really? Turing-complete markup? Such a big temptation, such a rich source of CVEs over the past decade...

Anyway. I think both problems can be circumvented by pregenerating HTML pages from Pod during upload preparation. To avoid polluting the source trees with the generated HTML, it could be pushed to a different repository. Or to a docs-only branch (on GitHub, it's called "gh-pages" and has URLs).

[zoffixznet]

Anyway. I think both problems can be circumvented

I don't see how that advice help at all with the topic of this Issue. Who'd be pregenerating anything? GitHub?

[jonathanstowe]

@zoffixznet off the top of my head I guess that a restricted setting could be made that only does enough to parse and render the pod, but it would still be somewhat vulnerable to anything in BEGIN, INIT or CHECK blocks.

[JJ]

And I would say this one is actually solved, right?

[AlexDaniel]

Not at all. For example, here's a random page that is still not rendered: https://github.com/perl6/doc/blob/master/doc/Language/about.pod6

As an example, here is a page that is rendered: https://github.com/perl6/nqp/blob/master/README.pod (but it's POD5)

[JJ]

I do have an issue with the external label. If it's something that has been reported elsewhere, it might be better to reference it here. If it's simply not related to this repo, we should simply close it. If we can fix it by working around the documentation quirks and rewrite the POD6 code, we should simply create a list of the pages affected and work through it, removing the external label.

[Skarsnik]

So does .pod6 get finally rendered correctly?

2018-01-27 20:39 GMT+01:00 Juan Julián Merelo Guervós < notifications@github.com>:

I do have an issue with the external label. If it's something that has been reported elsewhere, it might be better to reference it here. If it's simply not related to this repo, we should simply close it. If we can fix it by working around the documentation quirks and rewrite the POD6 code, we should simply create a list of the pages affected and work through it, removing the external label.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/perl6/doc/issues/167#issuecomment-361009595, or mute the thread https://github.com/notifications/unsubscribe-auth/AMYVf0XbMcUvEom9BwKWP6XMS-cpP9d7ks5tO3tugaJpZM4GQo3S .

-- Sylvain "Skarsnik" Colinet

Victory was near but the power of the ring couldn't be undone

[JJ]

@Skarsnik apparently not, but that's external to this repo. except if you want to change content to avoid that.

[AlexDaniel]

So does .pod6 get finally rendered correctly?

No.

As for closing tickets for issues that are still relevant, let's please not. The problem is still there, let's have a ticket for every problem. Otherwise people may open new dup tickets because, well, who would search through the closed ones?

[JJ]

@AlexDaniel this issue depends on another issue in the github/markdown (mentioned above) which does not seem like it's going to be closed or solved any time soon. In fact, googling perl6 pod rendering github returns that issue, not this one. There's slim chance someone else will open another ticket here on that particular matter. I think we can safely close this one. If some duplicate arrives, I'll close it immediately too.

[JJ]

And I'm closing this one, since nobody objected to this comment two days ago.

[AlexDaniel]

Uh… well, just because nobody objected doesn't really mean that it's right :)

There's also my comment above, where I think I've said it all already. We have external tag, and marking an issue as such is enough. Closing it does no good, having it open does not hurt and helps a lot when you need to find something.

[stmuk]

Has anyone experimented with rendering pod6 from perl 5?

Pragmatically it may be easier to get github to install a Perl 5 module than Perl 6 itself.

This would at least be progress (although clearly using Perl 6 would be better).

[JJ]

I do have an issue with external tags in general. They tend to stay there like for ever. Even after the original issue has been solved. With respect to this one, I don't think it's going to ever be solved. Pod6 parsing is included in perl6, and the other way round, it needs to include a perl6 parser. There's slim chance it's going to be written in any other language, and due to security concerns, that it's going to be included in GitHub. Just check the comments above and the comments in the GitHub markdown issue, also mentioned above. It serves no one, helps no one to have an issue that's not going to be solved any time soon, or maybe never, sitting pretty for 2+ years. I would like to have this kind of issues not only reopened, but also self-assigned and with someone committed to solving it or at least tracking what's going on in the other side. That, however, applies to other issues, NOT this one, which will never be solved. And that's why I have closed it.

[JJ]

2018-03-12 9:11 GMT+01:00 Aleks-Daniel Jakimenko-Aleksejev < notifications@github.com>:

Uh… well, just because nobody objected doesn't really mean that it's right :)

There's also my comment above, where I think I've said it all already. We have external tag, and marking an issue as such is enough. Closing it does no good, having it open does not hurt and helps a lot when you need to find something.

Having it open hurts when you try to set aside some time to solving issues and need to establish priorities. Going into it, seeing it marked as "external", meaning "it's somebody else's problem, so we can do zilch about it, but let's leave it here anyway", wastes my time. Keeping it in the tally of unsolved issues is not good image either. So it does hurt. It hurts me, if no one else. And when you need to find something, if google is used, it's going to search into solved and unsolved issues. It's not going to be more helpful if it's got a green badge saying "Hey! I'm not helpful and don't solve anything, but at least I'm green, meaning I care". My point is that issues in a particular repository should result in actionable changes in that particular repository. Should be close-able with a commit. That's not peculiar to this repo. It's the same for any repo. "external", by definition, can't be so. There's no commit in this repo (or in another, for that matter) referring to this issue. There is a closed issue, and a merged PR, time ago, and that's it. This means that the rest of the repo improvements can safely proceed without this particular one open.

EDIT: formatting

[AlexDaniel]

I do have an issue with external tags in general.

Okay.

They tend to stay there like for ever.

Just like other issues that are hard to fix.

Even after the original issue has been solved.

This is not specific to external issues. Many of our old rakudo tickets were like this.

I don't think it's going to ever be solved

Your pessimism is irrelevant to the fact that we have this problem that needs to be resolved.

I would like to have this kind of issues not only reopened, but also self-assigned

We are volunteers. Feel free to self-assign.

wastes my time

I'm very sorry. FWIW you can include -label:external in your search to ignore these tickets. There are only 3 of them at the moment though.

Keeping it in the tally of unsolved issues is not good image either.

Closing issues that are not resolved creates an image that is even worse.

if google is used, it's going to search into solved and unsolved issues

Are you saying that I should search for tickets using google? That's not how I've been doing things, but that's an interesting idea…

"Hey! I'm not helpful and don't solve anything, but at least I'm green, meaning I care"

That sounds pretty good to me? EDIT: to clarify, caring about something is the first step to getting something helpful, and eventually solving the issue

My point is that issues in a particular repository should result in actionable changes in that particular repository.

In an ideal world, yes. But sometimes we depend on other things and need to track progress and stuff. FWIW here's an example of an issue that needs no commits in the repo but heavily depends on other stuff. According to your logic that issue should not exist, even though it was (and still is!) super helpful.

This means that the rest of the repo improvements can safely proceed without this particular one open.

The rest of the repo improvements can safely proceed with this particular one open.

Sorry I don't think I want to discuss this matter more.

EDIT: FWIW all references in irc log to this issue: poink

[JJ]

Pinging again to see if there's anything missing from the PR. It looks like I've done all changes requested, but...

[JJ]

Just as an update, there was an attempt to implement that in GitHub, but it failed since, for the time being, there is no way to parse the documents and avoid running code. There was a test repo here, but it has been since abandoned. So I'm closing this, since it seems impossible with the current (and foreseeable) state of things.

[AlexDaniel]

… Yeah, but the issue is still not resolved, right?

[JJ]

It's never going to be solved. I see no point in keeping it open. But, whatever.

[AlexDaniel]

It's never going to be solved.

I haven't followed this issue, but can we stop this bullshit, please? Surely you can write a dumb parser that doesn't execute any code. How many of the actual .pod6 files need code to be executed in order to render them properly? Yes, of course a parser like that will fail to render some files, so what?

[AlexDaniel]

In fact, you have to execute code in order to do correct syntax highlighting. Guess what, it doesn't stop all of the editors to highlight perl 6 code.

[JJ]

Well, maybe you should have. Since the Pod6 parser is part of the perl6 parser, it's basically impossible to guarantee that pod parsing is not running any code. Check this answer in StackOverflow, for instance. What I'm telling above is the actual sequence of events, which I have been following (and that's the reason why I'm closing this). Please check this comment by Kivikakk, who was in charge of implementing this for GitHub, and their subsequent quitting. Baseline is: GitHub is not going to look at this any more. We can keep the issue open, close it or whatever. That's not going to change unless we come up with a Pod-only parser, which might or might not happen, but that will mean we will have to start the process all over again.

[AlexDaniel]

That's not going to change unless we come up with a Pod-only parser

OK, sounds great, let's keep this open then.

that will mean we will have to start the process all over again

Makes sense. We can also create a new ticket for our next iteration.

[JJ]

It would be much better if we closed this issue and summarized the problems for whatever comes next in a new one. Lots of people (including me) have wasted a lot of energy to solve it, for no good. It can be said that it has contributed to the greater good of Pod::To::HTML rendering, which it has, but still there's some specific work done to solve the issue which has gone to waste, and for precisely the reasons that were advised in several comments in this issue some time ago.

[JJ]

In fact, I am going to call a vote on closing this issue, for these reasons.

1. It's not actually a documentation issue, so it does not belong in this repo to start with. Rendering pod6 in GitHub might be a marketing, or an ecosystem issue, but definitely not part of the documentation.
2. The issue has actually been solved as posted. The pull requests created by several of us have actually been merged. They have finally been rejected in the actual production phase, but the work requested has actually been done.
3. The underlying problem (pod6 is parsed by perl6) is still there, and it's not going to be solved in terms of changes to this repository, so it does not belong to this repository. It might need a Pod6::Simple module in the ecosystem which has a Pod6-only grammar, or a Pod6::Clean module which cleans all code from the Pod before rendering it. None of them are, or will be, part of this repository.

So please vote by thumbs up (closing the issue), thumbs down (keep it open). And, of course, comment to explain your vote if you so like.

[AlexDaniel]

Ticket moved: https://github.com/perl6/user-experience/issues/35

Please don't close still-unresolved issues without creating a new ticket first.