Need help with understanding JWPUB format

[MrCyjaneK]

I have no idea how to get words out of Content in .db file located in jwpub archive. what I know. So any help is needed.

[geimist]

Hello @MrCyjaneK, sorry for my question, which is a bit off topic here, but because you wrote that:

5 days of interrupted work and I've imported most of JWPUBs that were easy to obtain (curl + jq in a for loop). I have managed to downlaod 414.89 GiB of them, and discovered few interesting things on my way

I want to keep a local JWPUB archive. Do you have any information on how you were able to load the entire catalog (I only want it in one language – that's only ~4 GB).

I am using the API url https://b.jw-cdn.org/apis/pub-media/GETPUBMEDIALINKS?issue=... so far. However, this must be structured differently for different publications (e.g. for periodicals), which I currently influence manually. In the future, however, I would also like to handle updates and new publications automatically.

How have you implemented this? (I only do this in the shell)

Thanks 🤗

[MrCyjaneK]

@geimist you need catalog.db

1. https://app.jw-cdn.org/catalogs/publications/v4/manifest.json
2. https://app.jw-cdn.org/catalogs/publications/v4/{current from manifest.json above}/catalog.db.gz

gzip -d it and enjoy sqlite catalog :)

[geimist]

Yes, I have that. But how do you find/create the download links? Or is there perhaps a way to initialize the download directly via the value Signature (ID of the publication) of the PublicationAsset table?

[GuyMicciche]

I have a working python version.

[MrCyjaneK]

@geimist - provide all parameters that you find to the url, if something is undefined provide an empty string.

[MaxBas70]

I have no idea how to get words out of Content in .db file located in jwpub archive. what I know. So any help is needed.

I would like to know if there is a way to convert .doc (win word) or pdf files into .jwpub format, or if there isn't a system already done online, an idea on how to do it

[MaxBas70]

I have no idea how to get words out of Content in .db file located in jwpub archive. what I know. So any help is needed.

I would like to know if there is a way to convert .doc (win word) or pdf files into .jwpub format, or if there isn't a system already done online, an idea on how to do it

[arthurwweber]

I have no idea how to get words out of Content in .db file located in jwpub archive. what I know. So any help is needed.

I would like to know if there is a way to convert .doc (win word) or pdf files into .jwpub format, or if there isn't a system already done online, an idea on how to do it

Yes, MaxBas70. There is a way. You have to create the SQLite database that defines the publication (with all its content, metadata, words index etc.), add the artwork as separate files, pack everything together, attach some more required metadata, pack it once more and there you have the JWPUB file. Since the JWPUB file is not a publicly documented format (most of us here have looked into JWPUB files to see how they are structured without any support from brothers serving in MEPS Programming), there isn't any publicly available piece of software that takes a random Word document and converts it into JWPUB.

[MaxBas70]

Thanks Arthur, at this point I ask you if it was possible to ### create an online converter that launches a procedure for the conversion. I know many who would be interested and would be willing to pay a monthly or processing fee to have, for example, speech outlines directly on JWL. I can provide the technological structure to do it and the marketing to sell it. If you give me a hand, if it's possible.

[arthurwweber]

Aren't some of the public talk outlines already available in JWPUB on JW Hub?

[MrCyjaneK]

it is already rather in depth explained in this very issue, I suggest reading all of the comments in here.

[livrasand]

I'm not sure if creating a "JWPUB online converter software" is legal to charge money. You won't find a direct WORD or PDF to JWPUB converter. This specific conversion can only be done through MEPS. MEPS provides an interface similar to that of WORD and allows the creation of digital publications. After you finish publishing and sending it to print, you can export it in formats such as PDF, WPUB, EPUB, and JWPUB. Have you already tried Reviw and its wiki? It can help you create a JWPUB easily and for free.

https://github.com/livrasand/Reviw

[MaxBas70]

Alcuni schemi dei discorsi pubblici non sono già disponibili in JWPUB su JW Hub?

Not all, some are in pdf or word. The problem is that if I download a pattern, I am forced to use it pure and make the notes alongside. Instead it would be more useful to upload my word or pdf outline, which I customized and have it in the main window. Or it would be useful to have a system to position the free text box as you like as in the W study. Can anyone explain to me how to use Rewind in practice?

[Fuseteam]

hey guys, have you seen this sections of jw.org/en/terms-use? "the Application" referring to JW Library

[gokusander]

Good evening, I'm creating a custom .jwpub, but I'd like some help with the .db tables. There are some that I don't know what to put: sqlite_stat1, word, searchindex, DocumentParagraph, and many others. I already have the html, manifest and blob, but it doesn't open. I believe there's something missing. I was told it would be the paragraph, but I don't know how to fill it in. If you can help, please. Thank you.

I can share the .zip for help

(I use google translator)😂

[goodniceweb]

Hello there.

I'm working on a service that helps with a personal study and I also need help / guidance around how to make my own JWPub file.

Idea of my service is the following: save your writings while listening on a congregation / assembly or while listening a video in the service. It splits your writings with AI into pieces so you can have your own index: when you read The Bible and tap on the gem icon, you can see part of your writing, tap on it and see the whole writing - of a speech or a video.

My progress so far:

• Encoding / decoding of a binary format for the Content column according to this guide - Done ✅
• Filling Publication table with metadata of my custom JWPub file - Done ✅
• Filling PublicationYear table with current year - Done ✅
• Filling PublicationCategory table with index - Done ✅
• Filling Document table with user's writings - Done ✅
• Filling Extract table with pieces of user writing split by AI - Done ✅
• Creating a zip file of an SQLite db file - Done ✅
• Creating a manifest file of JWPub file - Done ✅
• Creating a zip file which is actually JWPub file - Done ✅
• Fill RefDocument table - In Progress 🟡
• Fill VerseCommentary table - In Progress 🟡
• Fill VerseCommentaryMap table - In Progress 🟡
• Being able to install this file in JWLibrary - Not done 🙁 And I'm not sure what else I can do about it. Below you can find more details about what was done.

Details

• got SQLite file of the latest publication index from our official website. Unzipped it and removed all data from all tables, left just empty tables. Run vacuum on it so it release space on my SSD. So I think the structure of db should look good.

• Fill Publication table with the only record about my own index.

  See code and comments

  ```ruby publication_data = { PublicationId: 1, VersionNumber: 1, Type: 9, # Just didn't change what it was Title: name, # Name here means what user assign it to on the service TitleRich: nil, RootSymbol: SYMBOL, RootYear: Date.today.year, RootMepsLanguageIndex: 1, ShortTitle: name, ShortTitleRich: nil, DisplayTitle: name, DisplayTitleRich: nil, ReferenceTitle: name, ReferenceTitleRich: nil, UndatedReferenceTitle: name, UndatedReferenceTitleRich: name, Symbol: SYMBOL, # this is my own made-up symbol: mrsrch (stands for my-research) UndatedSymbol: SYMBOL, UniqueSymbol: SYMBOL, EnglishSymbol: SYMBOL, UniqueEnglishSymbol: SYMBOL, IssueTagNumber: 0, IssueNumber: 0, Variation: "", Year: Date.today.year, VolumeNumber: 0, MepsLanguageIndex: MEPS_LANGUAGE_INDEX, # I'm native Russian/Ukraine person so this is 207 for me PublicationType: PUBLICATION_TYPE, # "Index" string here PublicationCategorySymbol: CATEGORY_SYMBOL, # "dx" BibleVersionForCitations: "NWTR", HasPublicationChapterNumbers: 0, HasPublicationSectionNumbers: 0, FirstDatedTextDateOffset: 0, LastDatedTextDateOffset: 0, MepsBuildNumber: MEPS_BUILD_NUMBER # 13_073 } ```

• PublicationCategory and PublicationYear tables are self-explanatory but still gonna list them below

  See code

  ```ruby publication_category_data = { PublicationCategoryId: 1, PublicationId: 1, Category: "dx" } publication_year_data = { PublicationYearId: 1, PublicationId: 1, Year: Date.today.year } ```

• Then I loop through writings user created in my system and create Document records in the database.

  See code and comments

  ```ruby document_data = { DocumentId: document_array_index, PublicationId: publication_data[:PublicationId], MepsDocumentId: document_array_index, MepsLanguageIndex: MEPS_LANGUAGE_INDEX, Class: 3, # means regular I guess Type: 1, # playing guessing game SectionNumber: nil, # do not use sections for now ChapterNumber: nil, Title: writing.title, TitleRich: nil, TocTitle: research.title, TocTitleRich: nil, ContextTitle: nil, ContextTitleRich: nil, FeatureTitle: nil, FeatureTitleRich: nil, Subtitle: nil, SubtitleRich: nil, FeatureSubtitle: nil, FeatureSubtitleRich: nil, # Using this algo https://github.com/MrCyjaneK/jwapi/issues/1#issuecomment-1714309559 Content: encryptor.encrypt(writing.content.body.to_html), FirstFootnoteId: nil, LastFootnoteId: nil, FirstBibleCitationId: nil, LastBibleCitationId: nil, ParagraphCount: writing.paragraph_count, HasMediaLinks: 0, HasLinks: 0, FirstPageNumber: nil, LastPageNumber: nil, ContentLength: writing.content.body.to_html.size, PreferredPresentation: nil, ContentReworkedDate: nil, HasPronunciationGuide: 0 } ```

• Then I create records in the Extract table so users can see previews of their writings

  See code and comments

  ```ruby extract_data = { ExtractId: extract_index, Link: nil, # ToDo: find a way to set link. I think it's related with content of VerseCommentary->Content Caption: research.title, CaptionRich: nil, Content: encryptor.encrypt(extract.content), RefPublicationId: publication_data[:PublicationId], RefMepsDocumentId: document_data[:DocumentId], RefMepsDocumentClass: document_data[:Class], RefBeginParagraphOrdinal: nil, RefEndParagraphOrdinal: nil } ```

• Then I create zip out of db file. In the zip file, I put file inside of directory with the same name as the file, except the "*.db" extension. As I don't have images for this, I don't add them.

• I create manifest file.

  See code and comments

  ```ruby manifest_data = { name: "mrsrch.jwpub", hash: sha256of_zipped_db_file, timestamp: Time.now.utc.iso8601, version: 1, expandedSize: File.size(path_to_unzipped_db_file), contentFormat: "z-a", htmlValidated: true, mepsPlatformVersion: 2.100000, mepsBuildNumber: MEPS_BUILD_NUMBER, publication: { fileName: "mrsrch.db", type: 9, title: name, shortTitle: name, displayTitle: name, referenceTitle: name, undatedReferenceTitle: name, titleRich: name, displayTitleRich: name, referenceTitleRich: name, undatedReferenceTitleRich: name, symbol: SYMBOL, uniqueEnglishSymbol: SYMBOL, uniqueSymbol: SYMBOL, undatedSymbol: SYMBOL, englishSymbol: SYMBOL, language: MEPS_LANGUAGE_INDEX, hash: sha1_of_unzipped_db_file, timestamp: Time.now.utc.iso8601, minPlatformVersion: 1, schemaVersion: 8, year: Date.today.year, issueId: 0, issueNumber: 0, variation: "", publicationType: PUBLICATION_TYPE, rootSymbol: SYMBOL, rootYear: Date.today.year, rootLanguage: 0, images: [], categories: [ CATEGORY_SYMBOL ], attributes: [], issueAttributes: [], issueProperties: { title: "", undatedTitle: "", coverTitle: "", titleRich: "", undatedTitleRich: "", coverTitleRich: "", symbol: "", undatedSymbol: "" } } } ```

Then I zip zipped db + the manifest file into one "mrsrch.jwpub" file. And it can't be installed on JW Library at this point.

I feel a bit lost on what I can do next to make it work. Do you have any ideas? I'd appreciate any help 🙏

[livrasand]

Te recomiendo la wiki de Reviw: https://www.github.com/livrasand/Reviw/wiki its free

[goodniceweb]

Thanks I'll give it a try. Though, my Spanish to English translator sometimes gives me something crazy.

Maybe I'm missing something, but at least from what I understood, Reviw wants me to send them files so they create JWPub file for me, via GH issues. That's not how I want to approach the issue. But maybe I didn't understand their wiki.

[gokusander]

Thanks I'll give it a try. Though, my Spanish to English translator sometimes gives me something crazy.

Maybe I'm missing something, but at least from what I understood, Reviw wants me to send them files so they create JWPub file for me, via GH issues. That's not how I want to approach the issue. But maybe I didn't understand their wiki.

Is something like that, but Reviw i use to create a .db file, after that i create my own .jwpub (CyberChef i use for create my blobs). He helping with somethings i dont know, i believe he help you with your file and changing ideas.

That is my .jwpub, i'm working does 2 weeks and work well. He help me with something. So i believe is a good thing use there for asnwer your questions.

RefDocument table i create one, is complicated but cool.

I will add other talks, for now I have only made 1 Rename the file to .jwpub

am_T (1.0).zip

[livrasand]

Thanks I'll give it a try. Though, my Spanish to English translator sometimes gives me something crazy.

Maybe I'm missing something, but at least from what I understood, Reviw wants me to send them files so they create JWPub file for me, via GH issues. That's not how I want to approach the issue. But maybe I didn't understand their wiki.

If you need more specific help, I can help you. Me and the Reviw community have created all these JWPUBs: livrasand.github.io

Send me an email and we can talk about how to help you

[in-Load]

#

Thanks I'll give it a try. Though, my Spanish to English translator sometimes gives me something crazy. Maybe I'm missing something, but at least from what I understood, Reviw wants me to send them files so they create JWPub file for me, via GH issues. That's not how I want to approach the issue. But maybe I didn't understand their wiki.

Is something like that, but Reviw i use to create a .db file, after that i create my own .jwpub (CyberChef i use for create my blobs). He helping with somethings i dont know, i believe he help you with your file and changing ideas.

That is my .jwpub, i'm working does 2 weeks and work well. He help me with something. So i believe is a good thing use there for asnwer your questions.

RefDocument table i create one, is complicated but cool.

I will add other talks, for now I have only made 1 Rename the file to .jwpub

am_T (1.0).zip

hi everyone what's up ? Apologies, I'm French, I don't speak English and the translator is not so good to understand or learn, you know. Thank you so much to all of you for your contributions and repository, and thanks @gokusander for your testing file, but how you created the design interface (Ui) of your file please ?