Open Reqrefusion opened 2 months ago
chennes
commented
2 months ago Since you are not yet asking for a specific funding allocation, this proposal cannot yet be forwarded to the FPA Grant Review committee. I suggest you work directly with the members of the FPA to further develop this proposal and come up with a funding request. The FPA administrative team has virtual meetings every Wednesday, and can also be reached via email at fpa@freedcad.org.
Reqrefusion
commented
2 months ago Since you are not yet asking for a specific funding allocation, this proposal cannot yet be forwarded to the FPA Grant Review committee. I suggest you work directly with the members of the FPA to further develop this proposal and come up with a funding request. The FPA administrative team has virtual meetings every Wednesday, and can also be reached via email at fpa@freedcad.org.
Then I demand the $3000 fee, which was also paid for the previous work. Some before the work started and some after it was finished. Of course, if that's what you mean.
chennes
commented
2 months ago Yes, thank you. Onto the timeline: it doesn't (ever) have to be exact, but can you give a general estimate? Are we talking weeks? Months? Years?
Reqrefusion
commented
2 months ago Yes, thank you. Onto the timeline: it doesn't (ever) have to be exact, but can you give a general estimate? Are we talking weeks? Months? Years?
This is the first time I'm doing something like this, excuse my ignorance. I mean for the application, not the other one. If there are no problems and everything happens on the first try, it can be finished in a month or even less. I don't see such a thing as possible. 3 months would be an optimal time.
yorikvanhavre
commented
2 months ago I generally like the proposal, several observations:
Reqrefusion
commented
2 months ago As you say yourself above, the content is the main, precious thing. The number one reason to move out of MediaWiki is to separate the contents from the platform. I still don't understand well in your proposal how that is achieved. If the content wil still live in a database and won't be separated from the platform, then we need a better documented solution to overcome that, as it is a central point
My proposal may not be fully understood because I wanted to show all the alternatives. I have actually started the preliminary prototyping, you can check it here. As you can see, the content is kept as a file and separated from mediawiki. Here, using mediawiki is still a preference not a must. In other words, other parsers can be used, but I want to preserve the advantages of mediawiki. According to this system, when a change is made in the wiki, it will be reflected in github, and in the same way, a change in github will be reflected in the wiki. In other words, we will preserve the advantages of both. And if we want, we can use another parser to remove the wiki. I am not in favor of using another parser. The APIs I will be using are very well documented. The solution here is actually to improve mediawiki. This is also very well documented. I still don't say I fully understand it.
How would the interaction with crowdin work?
From what I understand here, Crowdin can be applied to files written in wikitext. It works the same way as Markdown. Of course, it can also be associated with the mediawiki API.
I tried git-mediawiki but couldn't make it work with our wiki (too big? Too small server? It always times out in the middle)
It's something that comes from the size; there are currently 20,140 pages on the wiki. It's a matter of size; there are currently 20,140 pages on the wiki. For git-mediawiki, be careful with pages over 1,000.
I would like to see a better plan with each plugin... Would you copy the list over to this issue, and maybe add a bit more description?
I wanted to leave this topic in issues because it was already too long. I had already explained the important ones, most of the plugins are self-explanatory, but if you want additional explanations. Of course I can. Also, I'm not a little scared :D. The wikis I've worked on before weren't this big.
yorikvanhavre
commented
2 months ago If we can have the wiki contents in files, I think that's the most important, and using mediawiki or not as a "rendering engine" should be fine (provided we can solve the other issues of course). Curious to see more of your experiment and how that can work!
Reqrefusion
commented
2 months ago If we can have the wiki contents in files, I think that's the most important, and using mediawiki or not as a "rendering engine" should be fine (provided we can solve the other issues of course). Curious to see more of your experiment and how that can work!
No lie, I've been laughing at "rendering engine" for two days. Not because it's wrong, but because we'll be using mediawiki for exactly this purpose. I've also tried kiwix. You can find the link to our wiki's zim file from here. Unfortunately, it will be active for 7 days. It was an experience that left many questions in my mind. You can download Kivix from here. You can also use it as web browser an add-on. It looks very promising for offline mode.
chennes
commented
2 months ago @Reqrefusion Thank you for your application, and for your responses above: I've forwarded your grant application to the Grant Review Committee. They will discuss it, ask you any further questions they have here, and will then forward a recommendation on to the FPA. That part of the process typically takes about two weeks, followed by the official FPA vote, which takes another 1-2 weeks. I will let you know the committee's recommendation when it is ready, and provide their feedback to you.
kkremitzki
commented
2 months ago I'm in favor of this proposal. It addresses several of the long-standing sharp edges in our wiki experience, chiefly offline data use, bidirectional editing, and user registration, as well as containing some nice modernization aspects and search improvements.
Reqrefusion
commented
1 month ago Thank you for the information @chennes. I have made progress in the experiments I mentioned before. Now the repository here is updated every 15 minutes according to changes in the wiki. So technically we have a constantly updated git copy of the wiki. Of course, without the media. I will fix them soon. Commit History is a bit long, unfortunately I missed some things while experimenting. Actually, this system is quite good, but it is not synchronous. I told you to prepare a plugin using a webhook to handle this process. But since I didn't have access, I did it this way. I will try that method when I have access.
Thank you too @kkremitzki for your positive comment. As I promised @yorikvanhavre, I am preparing a more detailed report for these plugins. Actually, it was supposed to be finished, but it was postponed a little bit because I was sick. I can't wait to present it as soon as possible.
I'm writing this part later, but I don't think it will be a problem since I don't think what I wrote will be seen. As a result of my research, I found that the best method for translation is to connect to the API in the wiki, which I found to be the best method. According to this method, I will connect the translation plugin with crowdin. In this way, both the translation plugin and crowdin can be used at the same time. Of course, I will make improvements to the translation plugin first. I'm not sure if TTMServer, that is, translation memory, is working. Here Crowdin API integration is a bit of a challenging process. The committee may request that this be transferred to another grant request or be evaluated within this grant request. This is the only way Flawless can predict a transition. If you ask me, since we won't be able to offer good machine translations (API access is expensive and free plans are inadequate), plugins like TWP - Translate Web Pages are better instead.
yorikvanhavre
commented
1 month ago It is still not clear to me, in this proposal, if either:
For me that would be the big decision factor.
Reqrefusion
commented
1 month ago It is still not clear to me, in this proposal, if either:
* The wiki still works primarily with a database, and the contents are **exported** to files * The wiki **natively** works from filesFor me that would be the big decision factor.
This project primarily has two steps. The first step is to improve your wiki, which is probably less popular. The second step is a wiki that works on files of more interest.
Basically, when there is a change in the github repository, the project will instantly transfer it to the wiki, and if there is a change in the wiki, it will also transfer it to github. So yes, it works via files. However, while doing this, it imports these files into the database. So simply github and wiki are synchronized. Now they work both through files and databases. In other words, if desired, user login to the wiki can now be disabled, in which case users will continue to contribute to the wiki via github. My goal is to ensure that both options are available to all users. I mean, why would we really give up on one?
I don't quite understand what is meant by the wiki running natively. As I said above, if access to the wiki is closed and changes to the wiki are made only through files, won't the wiki work via files? Also, for now, I can only export because I don't even have a membership, let alone any authority for the wiki. I will probably synchronize the wiki from github when I get a membership. However, for synchronization from wiki to github, I need permissions to install plugins for the wiki because I need to access webhooks.
So, if you ask why this project is like this, I want to protect the wiki and at the same time fulfill everyone's wishes. I want everyone to contribute as they wish. Anyone who wants can contribute from the wiki, from github. If administrators want to have the right to choose, they can only log in from github or the wiki.
So now we have the right to choose. Although I do not approve of it, the wiki can be closed at any time in line with this project, and not a single data will be lost. Instead of parser.php, an alternative parser is used depending on where it will be used. I think having a choice at every moment is a much better thing than being confined to a single option.
It is not either one of them, but is both of them.
yorikvanhavre
commented
1 month ago Ok thanks for the explanation. That is now clear for me :sweat_smile: And indeed that would work I guess.
Reqrefusion
commented
1 month ago Ok thanks for the explanation. That is now clear for me 😅 And indeed that would work I guess.
I'm glad it became clear. I hope it's clear in a good way. I accept that, as was said at the meeting, there was no strong opinion against this project. There are several reasons for this. The first of these is that the subject is not interesting, who really thinks about the document provider. No amount of effort here will make FreeCAD come of age. So there is no WOW factor. It couldn't happen even if one wanted to, because what makes the documentation impressive is its depth and breadth. Secondly, the project does not foresee major changes. This prevents it from being equally interesting. We don't suddenly switch to the platform everyone loves, instead we maintain things that have been around for years. Thirdly, the project is based on PHP, a language that the community is not very fond of. This part is actually the main reason why the project started because, unlike most people, I learned the first programming with PHP instead of Python. However, rather than personal reasons, the fact that mediawiki is based entirely on PHP is a more important reason. However, I try to write tools that can be run locally, especially in Python. But it makes more sense to run it on a server or it is necessary to use PHP for codes integrated with the wiki. There is a discussion about this on the forum.
chennes
commented
1 month ago Could you please provide a one-paragraph (fewer than 200 words) synopsis of the proposed work for inclusion in the FPA vote?
Reqrefusion
commented
1 month ago Could you please provide a one-paragraph (fewer than 200 words) synopsis of the proposed work for inclusion in the FPA vote?
I hope it will be enough. I hope I haven't forgotten anything. Things like this make me very nervous. You can find it below.
In order to improve the documentation, the existing documentation provider, the wiki, will be maintained and the data will be stored as a file in addition to the database. While the development of the wiki will be through various plugins, the other solution will be by synchronizing the wiki and github simultaneously. As a result, anyone can contribute via github or via wiki. Ultimately, a Wikipedia-like experience is aimed for the wiki. The current wiki will be stored as a file and will be developed with various offline solutions. In addition, other problems related to documentation are somehow the subject of the project.
chennes
commented
1 month ago Thank you. The FPA Grant Review Committee was unable to come to a consensus on this proposal, so I have stopped that discussion there with no recommendation one way or the other made to the FPA. The FPA vote was posted today: that process typically takes 1-2 weeks to complete. I will keep you posted about the status of the vote.
Reqrefusion
commented
1 month ago The relevant repository is now running on the github to wiki. Since translation pages are closed for editing, they do not work either, but normal pages can be edited this way. It also has nice advantages like this. https://github.com/FreeCAD/FreeCAD/issues/16488 It is really fun to follow Yorik's steps, examining the existing things makes it easier to achieve the desired result by writing much less code. I will postpone the github app creation part, which I am not looking forward to at all, and pursue something more functional. I'll make a post about my progress here soon.
chennes
commented
2 weeks ago @Reqrefusion thank you for your proposal. The FPA has voted to fund this project, please send email to fpa@freecad.org to coordinate payment.
Reqrefusion
commented
1 week ago @Reqrefusion thank you for your proposal. The FPA has voted to fund this project, please send email to fpa@freecad.org to coordinate payment.
I sent an e-mail, and I also stated some of my questions via e-mail.
Proposal description
First and foremost, I would like to apologize for the length of this text, as it is longer than requested. Rather than a simple application, this has been prepared as a research report. Unlike previous attempts, MediaWiki has been thoroughly researched here. In fact, MediaWiki is a platform I was already quite familiar with before conducting this research. In the past, I have set up and contributed to various hobby wikis. This research primarily explores how the issues mentioned were resolved in the past. Solutions are compared, evaluated, and my own suggestions are provided. The resources used in this process include Sliptonic's discussion, the previous application, and the developer meeting held on September 14, 2024.
Maintaining MediaWiki is not difficult; it is simply something that belongs to a different world. It is designed to be used by people from different groups and to run even on the cheapest shared servers that only support PHP. Years of usage have led to solutions being created for every problem, often exactly as needed. The fact that MediaWiki is an older platform is not a disadvantage; on the advantage, it is an asset. This proposal also declares my intention to be the maintainer of this wiki.
MediaWiki has several significant advantages, such as widespread familiarity, a well-established system, and the ease of making changes. Switching to another platform would mean losing these benefits, and the migration process would be unnecessary. Therefore, improving the existing wiki is the most sensible course of action. Currently, the wiki has not received much attention, but the goal is to enhance it and provide a user experience on par with Wikipedia by giving it the care it deserves.
Additionally, I have proposed solutions to address the other complaints that were raised. In essence, my primary approach will be to show MediaWiki the love it needs.
Deliverables
First, I will apply the existing solutions that have been used to solve the problems in the current wiki. As can be understood from this, the project will not diverge from MediaWiki, and the issues will be resolved. I have already mentioned the parts related to extensions here.
Improving the search feature: There is a complaint regarding this. I am confident that the solution applied in Wikipedia will address all the requirements related to this issue.
Responsive design: Similarly, I have already outlined how to solve this fundamental issue here. If I become the maintainer, I will implement this solution.
Opening direct user registration: This is a major issue for the current wiki. Wikipedia's efforts in this area are beyond impressive—probably no platform deals with more vandals than Wikipedia. Accordingly, I will adapt the solutions implemented in Wikipedia to the current wiki. I have mentioned some of these solutions in the relevant issue.
Offline solutions: The highly requested Git feature is essentially desired for offline usage. First, I would like to propose the Collection extension, which allows users to create a custom book from wiki pages. You can see an example of this on the relevant page of English Wikipedia.
However, the primary project supported and maintained by the Wiki Foundation for offline reading is much more effective. Wikipedia's offline versions, which started with the Wikipedia CD project in 2003, led to the creation of Kiwix, a battle-tested software that allows offline access to even the current version of English Wikipedia. And yes, it really has seen battles. Due to its offline nature, versioning is also supported, and it works in practically any environment. As you might expect, this is not limited to Wikipedia but can be applied to all MediaWiki-based wikis. By using Zimit, the MediaWiki file can be downloaded for offline use.
Improving the editor: The wiki is designed for everyone to contribute, and tools have been developed with this in mind. One of the most impressive tools is probably the Visual Editor, which unfortunately is not available on the current wiki. I plan to add this to the wiki and implement other enhancements to make contributions easier.
Wiki mirroring: Before diving into the main topic, I want to address the mistakes of previous projects:
GSoC 2023 project: While it aimed at the desired goal, the core issue was the lack of focus on the content rather than the new documentation method. For such a large migration, the priority should be the correct transfer of content. The system should be built around the content because that's what truly matters. However, here the content was not the primary focus.
Yorik's documentation attempt: This was my main inspiration. However, Yorik was not fully aware of what he was dealing with. MediaWiki uses Wikitext, a unique markup language that is far more advanced than Markdown. Yorik primarily used Pandoc, which, as shown, does not produce identical results. Even with better parsers, moving away from something as flexible as Wikitext, which our wiki contributors are familiar with, would not make sense. Additionally, the translation tool we use, Crowdin, supports Wikitext. Regarding parsers, I would like to mention that the only parser that provides perfect results for Wikitext is MediaWiki's parser.php. Although work has been ongoing for over 10 years on Parsoid, a full transition has not been achieved.
A truly reasonable solution: I've previously explained why I love MediaWiki. One of the best things about it is that there is a solution for almost everything. For cloning a wiki as a Git repository, there is Git-MediaWiki, which allows users to create a local clone of the wiki and even make changes from it. It still works quite well. You can access the Git clone of the Stratum0 wiki here, and there is also an article on how it was obtained. Before presenting my own proposal, I want to point out that this system meets all the requirements. Given that the readers of this text are likely familiar with Git, this solution may even be easier for them than the one I will propose. This project is particularly impressive for individual use. The difference in my approach is that I propose a solution based on web services rather than Git.
Now, my solution: As I mentioned before, my solution is built around web services, specifically GitHub and MediaWiki APIs. The advantage is that it requires much less code and maintenance. MediaWiki uses Wikitext, a markup language created specifically for wikis. It is so advanced that converting it into the simpler Markdown is essentially impossible. Therefore, the Wikitext versions of wiki pages will be stored in a GitHub repository. When changes are made, they will be transferred to the relevant wiki page using a webhook and the MediaWiki API. This is one key difference from the previous solution—it uses a webhook instead of a cron job because we can write extensions within the wiki. Similarly, changes will also be transferred in the other direction, using the PageSaveComplete hook and GitHub API. This way, the transfer will be bidirectional, leveraging the advantages of both the wiki and Git. Given the size of the wiki, there may be potential issues, but I do not foresee significant problems. These issues would only arise if both GitHub and the wiki experienced changes on the same page, but both platforms have safeguards against such conflicts. Most contributions will likely happen on the wiki, as it will be improved.
GitHub could also be useful for managing translations. As previously mentioned, Crowdin could be used for translation, though investigating the migration of existing translations is necessary. My preliminary research hasn't yielded optimistic results. A viable solution could be to gradually add Crowdin translations as they are completed and lock those pages from further edits on the wiki. As the project progresses, these issues will become clearer.
The best part is that I have already implemented a system to demonstrate this solution. Using the API, the entire wiki or specific sections can be displayed in any desired location without making any changes to the API itself. With further adjustments, this system will be even more streamlined. MediaWiki excels in offering a solution for every problem, and here it shows again through its extensions and skins. All features of the wiki can be accessed via the API and displayed as needed. This system allows the wiki to continue its existence as an intermediary layer. You may ask, "Why not remove that layer?" Unfortunately, this is not feasible. The most advanced parser for Wikitext is integrated with MediaWiki, and separating it would complicate things more than simply using MediaWiki. I view this as a positive aspect, as it means our wiki will continue to exist, and those who wish to work within it can do so.
Since we are using bots, why not take it further? As I mentioned, these types of bots are used on Wikipedia to create articles. Similarly, we can generate automatically written articles, but this is a topic for further research.
If desired, the recommendations can be acted upon without taking my proposal into consideration.
Timeline
I don't see this task as merely installing extensions. I see it as showing love and care to the wiki. Additionally, one significant issue I cannot fully solve is the inability for everyone to view the code, which prevents me from giving an exact timeline for what I will be working on.
Risks and mitigation
I don't believe this work will be interrupted by anything I can foresee. Of course, I want to acknowledge that I cannot predict the future. However, I don't think the changes to be made are critical. If they were, the person currently handling it would have likely done them already.
But I can guarantee that this will be much faster than moving the wiki to an entirely new platform.
Compensation
Honestly, I don’t know the exact value of the work I’m going to do. I leave it to you to decide how my efforts will be compensated. The reason I’m making this proposal is that I believe we need to come to an agreement. This is because I will need certain additional permissions to do this work, and I will also have access to some confidential information related to the wiki. Furthermore, previous efforts have been made for this task, and they were compensated with a specific amount.
The main challenge of the system is the sheer number of components involved. I will be creating several tools outside the system as well. This is because much of the work will be handled via APIs, without requiring any conversion. The bulk of the work will be research and fine-tuning, and this applies to the system as well. I can already say that this research has likely provided a better understanding of the current situation than any previous efforts. I have proposed solutions that can be implemented even without the acceptance of my proposal.
About you
I am Turan Furkan Topak, and I use the nickname Reqrefusion. I’ve previously worked with MediaWiki on shared servers. Using only FTP, we managed to install some rather interesting extensions. I’ve also migrated a few wikis to my own custom software in the past. I was an active user of Wikipedia in my native language for a long time, and I’m quite proficient in PHP, as you can see from my past contributions. I’ve also designed technical drawing icons for FreeCAD and enjoy showing love to the current website.
This writing has gone beyond a simple proposal and evolved into more of a research report. However, there are likely things I’ve forgotten to mention, and I’ll do my best to answer any questions you may have.