search

kiwix / overview

:balloon: Start here for current projects, how to get involved with offline projects and joining community calls. A resource for new and veteran members
87 stars 14 forks source link

Nautilus Web UI #77

Closed rgaudin closed 12 months ago

rgaudin commented 1 year ago

This ticket tracks the Kiwix GSoC 2023 Project Nautilus Web UI until code contributions and/or specific tickets requires creating its own dedicated repository.

Candidates, contributors, this ticket is the preferred location to discuss this project. Ask your questions here and mentors (@rgaudin, @kelson42) shall respond.

Mandatory reads:

Nautilus Web UI

Objective: build a web-based interface for what is currently a command-line tool.

Technologies: Python, JavaScript, HTML, CSS

Description:

Kiwix is useful because it can provide a lot of content to people without internet access. Therefore, the easiest it is to generate content that is available offline, the more users can be reached.

Nautilus is a powerful tool for creating ZIM files from collections of documents (PDF files, videos, etc.), but it currently requires the user to run commands from the command line. This makes it challenging for users (usually schools) who are not familiar with the command line or who prefer a graphical user interface.

The goal of this project is to build a web-based user interface for Nautilus, making it easier for users to convert their collections of documents into browsable ZIM files. The web UI will be built using Python, JavaScript, HTML, and CSS and will provide an intuitive and user-friendly interface for Nautilus.

Key Deliverables:

Skills required:

Difficulty: Medium, but probably a 350hrs project

Rishabhg71 commented 1 year ago

Hi, I am Rishabh a 2nd year CSE student with 2-3 years of professional experience in web development who is looking to contribute to open source now.I have read the requirements that you have given but i have a really simple questions

rgaudin commented 1 year ago
  • Is tech stack already set in stone (i mean can it be changed like instead of JS is TS a possibility?)

Somewhat yes, because we'll have to maintain what will be built and we tend to limit the number of technologies we need across our projects. So most likely Python via Flask or FastAPI and the frontend using Vue.js

  • Will a single person work on this project or maybe a small team

Anybody can contribute to our projects but we'll have at most one GSoC student assigned to the project. We don't know yet how many student slots Google would grant us. This all depends on the quality of the candidates' proposals.

Rishabhg71 commented 1 year ago

Somewhat yes, because we'll have to maintain what will be built and we tend to limit the number of technologies we need across our projects. So most likely Python via Flash or FastAPI and the frontend using Vue.js

Do you mean Flask? or some library i am not aware of

rgaudin commented 1 year ago

Ah ah that's a typo, yes flask 😅

Rishabhg71 commented 1 year ago

Sorry for having so many questions 😓 but I want to ask will be it ok to post any questions here on this thread or is there any way to DM you Also since this project will be done from scratch? To make my candidature stand out should start contributing to other repos until then or try to make a Proof of concept/Prototype for this Idea? it could help in faster development time since i will be already aware of toolchains and tech

rgaudin commented 1 year ago

Here's the best place as any other candidate or contributor will have the opportunity to look at your questions and the answers. Saves times for everybody.

What's important to understand is that we get a lot of proposals and and more often than not what makes candidates stand out is contributions. We want to make sure that the candidate we choose is:

Contributions (to any repository of Kiwix, openZIM or Offspot) informs us abut all this.

m-raafat-abdelkader commented 1 year ago

Hi, I am Mohamed. I have good knowledge of python. And I currently work on being a Front-End Developer but I am still a newbie. My question is: Is the project suitable for a beginner at web development?

KrishnaRishi2208 commented 1 year ago

Hi, I am Rishi a 3rd year CSE student. I have read the problem statement and I am confident that I can finish this project as I have developed quite a few applications with the given tech stack.

The problem statement asks us to design a User Interface using Vue.JS as frontend and Flask at the backend providing an API for the Vue application.

My doubt is that , while developing the UI, how do we integrate Nautilus into our application. Are there any pre built API's that can help me take advantage of nautilus's functionality or do I have to clone the Nautilus repository locally and build my application on top of the Nautilus and develop my own API's to get Nautilus's functionality onto my application.

I was also thinking of allowing users to view zim files in kiwix directly from the application by redirecting the zim file to open in kiwix. Do you think that this is a good feature to implement and would it enhance that chances of my proposal being selected?

Rishabhg71 commented 1 year ago

@KrishnaRishi2208 i think nautilus is just a term they are using for a File manager(i could be wrong)

KrishnaRishi2208 commented 1 year ago

@RG7279805 Nautilus is a software turns a collection of documents into a browsable ZIM File. It is already developed , but right now the usage is only through command line. So, the purpose of this project is to develop a UI on top of the existing software.

hodzy commented 1 year ago

Hello Team, My name is Hady Elkady, a fresh graduate from the university of Bonn in Germany, where i have studied my bachelors, I currently work with a research group at the university of Oxford, where i have been developing a website to host the datasets offered by the group using Django and vanilla javascript.

previously i have worked during my studies in multiple projects and with another research group using python, if needed, i can gladly send my cv.

Regarding the project, I wanted to ask you as this would make development faster in my case, would it be ok to use django instead of flask, and what level of experience do you guys expect from developers before joining.

Also wanted to ask if there are any challenges or a mock up that we would need to finish before joining?

thank you in advance

Rishabhg71 commented 1 year ago

@RG7279805 Nautilus is a software turns a collection of documents into a browsable ZIM File. It is already developed , but right now the usage is only through command line. So, the purpose of this project is to develop a UI on top of the existing software.

Ahh my bad but after your clarification its clear now,thanks

Now moving to your question i think we have to clone the current nautilus repo and then build a flask app upon that by importing already written functions (I could be wrong but want to discuss possible solutions)

hodzy commented 1 year ago
  • Is tech stack already set in stone (i mean can it be changed like instead of JS is TS a possibility?)

Somewhat yes, because we'll have to maintain what will be built and we tend to limit the number of technologies we need across our projects. So most likely Python via Flask or FastAPI and the frontend using Vue.js

  • Will a single person work on this project or maybe a small team

Anybody can contribute to our projects but we'll have at most one GSoC student assigned to the project. We don't know yet how many student slots Google would grant us. This all depends on the quality of the candidates' proposals.

@rgaudin @kelson42 well my idea was, using django, as the more famous when developing large full stack software, would benefit the project quality and possibly performance, and considering it is also a python framework, it shouldn't affect the development, when imoprting already written python functions

And i wanted to know if there is a challenge that we have to finish before submitting our proposal, or do we have to commit to for example to the repo before the beginning of the program?

KrishnaRishi2208 commented 1 year ago

@RG7279805 Nautilus is a software turns a collection of documents into a browsable ZIM File. It is already developed , but right now the usage is only through command line. So, the purpose of this project is to develop a UI on top of the existing software.

Ahh my bad but after your clarification its clear now,thanks

Now moving to your question i think we have to clone the current nautilus repo and then build a flask app upon that by importing already written functions (I could be wrong but want to discuss possible solutions)

Thanks for the clarification . I think we have to import and use Nautilus functions to get the required functionality.

Rishabhg71 commented 1 year ago
  • Is tech stack already set in stone (i mean can it be changed like instead of JS is TS a possibility?)

Somewhat yes, because we'll have to maintain what will be built and we tend to limit the number of technologies we need across our projects. So most likely Python via Flask or FastAPI and the frontend using Vue.js

  • Will a single person work on this project or maybe a small team

Anybody can contribute to our projects but we'll have at most one GSoC student assigned to the project. We don't know yet how many student slots Google would grant us. This all depends on the quality of the candidates' proposals.

@rgaudin @kelson42

well my idea was, using django, as the more famous when developing large full stack software, would benefit the project quality and possibly performance, and considering it is also a python framework, it shouldn't affect the development, when imoprting already written python functions

And i wanted to know if there is a challenge that we have to finish before submitting our proposal, or do we have to commit to for example to the repo before the beginning of the program?

I think the answer you are looking lies in your question You have mentioned that django is suitable for large scale apps But you look at the nautilus CLI you will see it bot that huge so Flask might be a better option since we just have to expose the CLI features via HTTP (I could be wrong and maybe mentors might change stack )

hodzy commented 1 year ago

although i would have appreciated if we can give the mentors the chance to answer the question, as they are the ones who are more knowledgeable of the codebase, and of the flexibility of the project, and won't just guess from my question.

@RG7279805 But really thank you for your answer, and being known for being used for large scale projects bec. of the middleware available, doesn't mean it can't be used for small projects, and i assume you haven't used django before, but it is not really difficult to setup for small projects and even the first website i have seen that compares the 2, states that it can be used for simple websites.

hope the mentors can still answer my question

rishav-jha-mech commented 1 year ago

Hi I am Rishav Jha, a 3rd year Bachelor of Technology student from National Institute of Technology Durgapur. I am an experienced Full stack developer and have worked on many Projects in Python. I have worked on many projects involving Management Systems, I have also worked and created several custom websites using the beautifulSoup and youtubeDL package, this make me enthusiastic about working on this project.

If we will be using Django/ Flask then we can use its templating engine to render the files but testing the frontend code will be harder, but since we are creating APIs. I think using FastAPI will be the best option for this project, because it has built-in support for async code, and automatic API documentation generation, which will speed up our development time, make the project developer friendly, testing will be way more easier.

But I need to ask one thing, can we work on React for the UI part ?, because it has a large active community and thus maintaining the project in the longer run will be easier, because a lot of developers already know React. To test the Web UI we can use Jest & Enzyme.

My Github Account - https://github.com/rishav-jha-mech My Portfolio - https://rishav-jha-mech.github.io/devraj

Arpitkandwal commented 1 year ago

Hello Everyone! I am Arpit kandwal, Software Engineering Student from India. I'm excited to join this community and contribute to GSoC program. I have some experience in javascript, Nodejs, git and MongoDB. I have Created a website Using MongoDB, Expressjs, Nodejs. My works and projects is mentioned in my github. My github Profile is: https://github.com/Arpitkandwal

rgaudin commented 1 year ago

Thank you all for your feedback ; it's exciting to see this much interest in this project. Saturday/Sunday are my AFK days so pardon the delay. I will respond to all questions in this single comment.

@m-raafat-abdelkader

Is the project suitable for a beginner at web development?

Yes ; a good part of GSoC is for the student to learn by applying their skills to an actual project to gain experience. That said, as mentioned above and on GSoC website, first step is for organizations to pick a single student per project. I've mentioned our criteria above yet you can imagine students with stronger background tend to have more chances.

@KrishnaRishi2208

My doubt is that , while developing the UI, how do we integrate Nautilus into our application. Are there any pre built API's that can help me take advantage of nautilus's functionality or do I have to clone the Nautilus repository locally and build my application on top of the Nautilus and develop my own API's to get Nautilus's functionality onto my application.

nautilus is a python app that only has a CLI interface. It has not been designed to work with this webUI so it might be necessary to tweak it a bit (part of the project). As to how technically call it, I'd say via Python. The package can be installed via pip.

I was also thinking of allowing users to view zim files in kiwix directly from the application by redirecting the zim file to open in kiwix. Do you think that this is a good feature to implement and would it enhance that chances of my proposal being selected?

It's a very interesting idea indeed but there might be some challenges.

@RG7279805

@KrishnaRishi2208 i think nautilus is just a term they are using for a File manager(i could be wrong)

Yeah we chose the name Nautilus because of the Nautilus File Manager… because nautilus's source of data is files. The two projects aren't related at all.

@RG7279805 Nautilus is a software turns a collection of documents into a browsable ZIM File. It is already developed , but right now the usage is only through command line. So, the purpose of this project is to develop a UI on top of the existing software.

Exactly.

@hodzy

Regarding the project, I wanted to ask you as this would make development faster in my case, would it be ok to use django instead of flask, and what level of experience do you guys expect from developers before joining.

There's no expected level of experience.

As for Django, no, we want to limit the number of technologies we use across our organization to be able to follow them and maintain our projects.

Also wanted to ask if there are any challenges or a mock up that we would need to finish before joining?

No, please refer to GSoC instructions. At this stage, candidates should be participating in our orgs' repository to demonstrate their existing skills. Later this month, candidates will have to submit a proposal that will be the base of the candidate selection.

@rishav-jha-mech

But I need to ask one thing, can we work on React for the UI part ?, because it has a large active community and thus maintaining the project in the longer run will be easier, because a lot of developers already know React. To test the Web UI we can use Jest & Enzyme.

No, as explained above, we'll be maintaining this so we want to use the technologies we are already using. It will probably be up to the candidate to choose but it will Flask or FastAPI and either backend-rendering or Vue.js for the frontend.

Waer1 commented 1 year ago

hey Mr. Gaudin, I am Yousef Khaled Al-waer senior computer engineer student at Cairo university. I am interested in contributing to " Nautilus Web UI ", I am a DevOps engineer, I have almost 2 years of professional working experience in DevOps. I come from a development background. I worked as a frontend for about two years on a freelance, in addition to 6 months of working professionally.

basically, I understand the need for this project since I am one of the users of Kiwix, so I understand how it difficult how to treat with terminal if you aren't familiar with it, I would like to contribute to the project and in addition, I suggest dockerizing the whole application so the user can even run it inside the container when it needs to use without the need for installation, I want to contribute so is there is qualification task or some tasks I can do to prove my ability? something like an issue or place to discuss ideas.

thank you, Mr.Gaudin

Here are my contact details: E-mail Personal: elwaeryousef@gmail.com Github: https://github.com/Waer1/ Linkden: https://www.linkedin.com/in/yousef-elwaer/ Mobile: +201557752400

Yours Sincerely, Yousef Alwaer

rgaudin commented 1 year ago

@Waer1 indeed the whole application will be published via a docker image as well, as for all our projects.

Please read the previous comments regarding how to participate

Rishabhg71 commented 1 year ago

@rgaudin can you tell me the major target of this Nautilus Web UI. Is it gonna run on a small server to which people will have access or they will run it on the local machine serving content/files from their own machines

rgaudin commented 1 year ago

No, the main goal of this is to offer a SaaS like https://youzim.it for people to upload their files and download a ZIM file. It will also be able to run on anyone's machine (via Docker or Python) but that would not be the primary use case as the goal is to provide a U.I to simplify the process…

Rishabhg71 commented 1 year ago

@rgaudin I am ready with a first draft of my proposal could let me how can i share it with you can I DM this somewhere?

rgaudin commented 1 year ago

I cannot commit time to review proposals at this moment. You'll have to wait for the proposals window to open (the 20th I believe) and submit it via GSoC UI

kinxyo commented 1 year ago

Hey all, I'm Kinjalk Tripathi (2nd year B.Tech in CSE), I'm having trouble in setting up Nautilus in Windows. It always comes back to me with an error saying pyproject.toml did not run successfully: libzim library not found. After some googling I learnt that pip install libzim is not supported for windows and I will have to compile its c++ source code for that by installing additional dependencies, so I just wanted to ask, is Nautilus setup really this hectic or am I on a wrong track?

rgaudin commented 1 year ago

Nautilus depends on python-libzim indeed which is not supported on Windows. To use it on Windows, use it via docker

deepu-bansal commented 1 year ago

hey @rgaudin where, is the code or we are expected to make whole user interface . If so pls give some idea how we have to proceed

rgaudin commented 1 year ago

The whole UI is expected. Look at the inputs of nautilus ; those should be provided via the Web UI.

KrishnaRishi2208 commented 1 year ago

@rgaudin since the project is Medium but 350 hrs, can we divide it into 22 weeks or is there any specific time frame we have to follow while writing our proposal.

And , while preparing the proposal, is it better if we present basic HTML files we will use to design the UI or can we use tools like figma to develop the UI .

kartiksprajapati commented 1 year ago

Hello 👋,

I'm Kartik S Prajapati and I am pursuing my Bachelor's degree in Computer Applications from Chandigarh University in India. I am a Full Stack Web Developer with experience in using Vue.js, and I am excited to learn and work with Flask, which will be a new experience for me.

The problem at hand is with Nautilus, a tool developed in Python used to convert media files into ZIM file format. However, it is only available as a CLI and lacks a User Interface, making it difficult for those who are not well-versed with CLI to use it effectively.

To address this issue, We need to develop a User Interface using Vue.js as the frontend along with Tailwind.CSS for CSS. For the backend, We are gonna use Flask or FastAPI, both of which are Python frameworks. Right?

Have I correctly identified the requirements of the problem and proposed a suitable solution that aligns with them? Please let me know, and I would be happy to consider them. @rgaudin @kelson42

Thanks!

drtrain25 commented 1 year ago

Hi @rgaudin , I am John and I am currently pursuing my Bachelors in Computer Science.

While trying to use Nautilus locally, I have the following output in the cli : [nautiluszim::2023-03-09 09:40:07,887] INFO:starting nautilus scraper for testnaut.zip [nautiluszim::2023-03-09 09:40:07,888] INFO:preparing build folder at /output/build INFO:fixing videosjs-ogvjs.js INFO:all done. [nautiluszim::2023-03-09 09:40:07,924] INFO:Extracting ZIP archive /home/john/Desktop/testnaut.zip to /output/build/files [nautiluszim::2023-03-09 09:40:07,927] INFO:Collection loaded. 1 items, 1 files [nautiluszim::2023-03-09 09:40:07,927] INFO:update general metadata [nautiluszim::2023-03-09 09:40:07,945] INFO:creating HTML files [nautiluszim::2023-03-09 09:40:07,951] INFO:building ZIM file Resolve redirect set index [nautiluszim::2023-03-09 09:40:09,605] INFO:removing HTML folder [nautiluszim::2023-03-09 09:40:09,611] INFO:all done!

my collection.json file is: [ { "title": "...", "description": "...", "authors": "...", "files": ["./testnaut.zip"] }

]

My zip file has 3 empty text files: test1.txt,test2.txt,test3.txt

Could you please guide me in the usage of the tool as I cant see the zim file anywhere.

rgaudin commented 1 year ago

@KrishnaRishi2208, that's your proposal so you propose but it can all be discussed and changed afterwards. We don't expect code/html because it's just a proposal. Mockup images are a great idea!

@kartiksprajapati yes; I've already confirmed this multiple times.

@drtrain25 use --help to learn about the options. One of them is --output I believe

rgaudin commented 1 year ago

@shapaozi1hao, I've already answered all those questions. Also, you should communicate in English.

shafeipaozi commented 1 year ago

@shapaozi1hao，我已经回答了所有这些问题。此外，您应该用英语交流。

I'm sorry，I'm just worried that I don't have enough knowledge and hope to know more。

kartiksprajapati commented 1 year ago

Would request the mentors @rgaudin @kelson42 to please guide me on the next steps and project planning.

Thanks in Advance

rgaudin commented 1 year ago

@kartiksprajapati that's for you to figure out. Read the previous comments and GSoC documentation

itspallavi20 commented 1 year ago

Hello, I came across this project Nautilus Web UI and after reading the description and details I am getting that this idea is focused on providing the best Web UI via Nautilus to the users so that they will be able to download and upload ZIM files easily as UI will make it easier and convenient for users rather than command line . Please let me know if I am getting this right as I am highly motivated for being part of this project.

MadhuMaayera commented 1 year ago

Hi I am Madhurima from BTech 3rd year CSE branch.I am from India.Currently ,I am working on the skills required to be a part of your project.I will be highly obliged if you give me a chance to be a part and learning attitude inspires me that I can contribute something. contact details: https://www.linkedin.com/in/madhurima-sit-220627219

ngugimuchangi commented 1 year ago

Hello, everyone. I am Duncan Ngugi from Kenya. I've been studying SE for over a year now under the ALX-Holberton SE program. I have experience with C, Python (including Flask), HTML, CSS, and vanilla Javascript. I would really like to contribute to this project.

@rgaudin

No, the main goal of this is to offer a SaaS like https://youzim.it for people to upload their files and download a ZIM file.

  1. Does this mean that web UI should carter for the upload of zip files that nautilus can ZIM files
  2. Could, a kiwix browser extension/add-on be ideal to allow users to view zim files on their browsers, offering just reading functionality
rgaudin commented 1 year ago
  • Does this mean that web UI should carter for the upload of zip files that nautilus can ZIM files

Yes

  • Could, a kiwix browser extension/add-on be ideal to allow users to view zim files on their browsers, offering just reading functionality

Already exists: https://download.kiwix.org/release/

massifcoder commented 1 year ago

Respected Sir, I have created a flow chart representing what will be the flow chart of the process. Can you please verify that, am I taking the project idea correct or not? Or suggest any changes needed in it or not. Flow chart doing steps are:

Please suggest am I taking the idea right, so I can plan how to execute it in such a way that site will be more optimized. prop_page-0001

rgaudin commented 1 year ago
  • A UI of site, where website take the link of the site, whose zim file it wants.

No, this already exists https://youzim.it See its repo at zimit-frontend

We want something similar but not to create a ZIM from a website but instead from a collection of local files. Creating a ZIM from a collection of local files is what nautilus does but it's a CLI program that runs on unix so no Windows (except via docker or WSL). We thus want a SaaS to allow users to get a ZIM using just a browser and the server will call nautilus and upload the ZIM somewhere.

drtrain25 commented 1 year ago

@rgaudin I am working on a proposal for the project. Could you please clarify if we need to contribute to kiwix for our proposal to be considered for selection as I really want to get selected. If yes, could you please guide me by mentioning a few issues that I can work on.

rgaudin commented 1 year ago

You don't need to but I explained it's one of our criteria to select candidates. Read above comments.

santhos-13 commented 1 year ago

hello all, am santhosh currently pursuing electrical and electronics engineering in india.

is there any slack channel or any other medium for communication for this project? @rgaudin

rgaudin commented 1 year ago

#gsoc channel on Kiwix's Slack which mentions that questions would be answered only inside this ticket for now so that all candidates can see them

rohitmishra666 commented 1 year ago

Hello all, my name is Rohit currently pursuing mechanical engineering in India. are there any good first issues for me to start and help the codebase better? @rgaudin

rgaudin commented 1 year ago

Good first issues have a matching label in all our repositories

drtrain25 commented 1 year ago
  • A UI of site, where website take the link of the site, whose zim file it wants.

No, this already exists https://youzim.it See its repo at zimit-frontend

We want something similar but not to create a ZIM from a website but instead from a collection of local files. Creating a ZIM from a collection of local files is what nautilus does but it's a CLI program that runs on unix so no Windows (except via docker or WSL). We thus want a SaaS to allow users to get a ZIM using just a browser and the server will call nautilus and upload the ZIM somewhere.

@rgaudin You have mentioned that Nautilus is only available in windows using docker or wsl. If our application is running on windows, how do we call nautilus functions from our application? Will there be any cloud hosting of Nautilus where we can call the functions from flask or do we locally install nautilus and work on our application on top of it. If we will have to work on it locally , do we have to deploy our application by dockerizing our entire application ?