Upgrade news data model to note Part 1

azayati commented 5 months ago

Rationale

eXo is working on bringing it's full-featured news publishing system to meeds by leveraging Notes as content storage.

Changes need to be applied to notes prior bringing the News features.

1. Functional requirements

A News can be created even if there is no Notes app in a space
A News can be created even if there is the Notes app in a space but before the Notes app has been opened a first time
The notes draft are shared with users with edit permission. During migration for the notes with existing drafts, we will keep the latest edited draft only and share it.
The news draft are shared with users with edit permission. During migration for the news with existing drafts, we will keep the latest edited draft only and share it.

Update : The following requirement are moved in another MIP https://github.com/Meeds-io/MIPs/issues/130 :

migration activities
sheduled publication
following filters in news app : posted articles, published articles, my posted articles, archived articles, scheduled articles
in order to hide the options scheduled publication and filters, a flag is expected

2. Technical Requirements

Feature Flags

Property Name	Default Value	Target Audience	Functional Behaviour
newsScheduleAndFilterDisplaying	false	users	enable/disable new scheduling and filters in news app except draft filter

3. Software Architecture

Services & processing

Social addon

A new method getMetadataItemsByMetadataNameAndTypeAndObjectAndSpaceIds need to be implemented for metadata service layer
A new method getMetadataItemsByFilter need to be implemented for metadata API layers.

Notes addon

Currently only the "Home" note page can be created at the wiki space root level and any creation of another page at the same level updates the "Home" note page. In order to create a dedicated "news" note page at the wiki space root level, we need to allow the creation of any page at the wiki space root level by updating the method createPage of JPADataStorage layer (https://github.com/Meeds-io/notes/blob/develop/notes-service/src/main/java/org/exoplatform/wiki/jpa/JPADataStorage.java#L217).
A new method removeDraftById need to be implemented for notes API layers.
In ordre to implement the shared notes draft managment, we need to implement get draft methods without username for notes API layers

Content addon

A new meeds addon called content will be implemented:
- The rest and service layers of "Content" addon will be the same as eXo "News" addon rest and service layers.
- The jcr storage layer will be removed.
- All the jcr/ecms dependencies will be removed.
- The Content service layer will call the Notes service layer and the Metadata service layer.

Sequence diagram for create draft article use case:

Sequence diagram for get draft article use case

Data and persistence

A new metadataType "news" should be configured using org.exoplatform.social.metadata.MetadataTypePlugin component plugin.

    <component-plugin>
      <name>NotesMetadataPlugin</name>
      <set-method>addMetadataTypePlugin</set-method>
      <type>org.exoplatform.social.metadata.MetadataTypePlugin</type>
      <init-params>
        <value-param>
          <name>shareable</name>
          <value>true</value>
        </value-param>
        <value-param>
          <name>allowMultipleItemsPerObject</name>
          <value>true</value>
        </value-param>
        <object-param>
          <name>metadataType</name>
          <object type="org.exoplatform.social.metadata.model.MetadataType">
            <field name="id">
              <int>1000</int>
            </field>
            <field name="name">
              <string>news</string>
            </field>
          </object>
        </object-param>
      </init-params>
    </component-plugin>

A new metadata "news" associated to the metadataType "news" detailed above should be created into METADATA table using the configuration of org.exoplatform.social.metadata.MetadataInitPlugin component plugin. This metadata will be used in order to complete the WIKI_DRAFT_PAGES, WIKI_PAGE_VERSIONS or WIKI_PAGES fields with news additional fields.

    <component-plugin>
      <name>DraftNewsMetadataPlugin</name>
      <set-method>addMetadataPlugin</set-method>
      <type>org.exoplatform.social.metadata.MetadataInitPlugin</type>
      <init-params>
        <object-param>
          <name>metadata</name>
          <object type="org.exoplatform.social.metadata.model.Metadata">
            <field name="type">
              <object type="org.exoplatform.social.metadata.model.MetadataType">
                <field name="id">
                  <int>1000</int>
                </field>
                <field name="name">
                  <string>news</string>
                </field>
              </object>
            </field>
            <field name="name">
              <string>news</string>
            </field>
          </object>
        </object-param>
      </init-params>
    </component-plugin>

SOC_METADATAS table:

METADATA_ID	TYPE	Name	.....
5	1000	News	...........

When a news draft article, a news article or a news article version is created, a metadata item should be created into SOC_METADATA_ITEMS table. This metadata item will have:
- The "news" metadata id as metadata_id
- "draftPage", "page" or "pageVersion" as object_type, this will ensure to distinguish between metadata items with the same OBJECT_ID
- The corresponding notes draft_page_id, page_id or page_version_id as OBJECT_ID.

METADATA_ITEM table:

METADATA_ITEM_ID	METADATA_ID	OBJECT_TYPE	OBJECT_ID	PARENT_OBJECT_ID	...........
100	5	newsDraftPage	100	null	...........
101	5	newsPage	101	null	...........
102	5	newsPageVersion	102	null	...........
103	5	newsLatestDraftPage	103	101	...........

For the newly created metadata item, some properties should be created into METADATA_ITEM_PROPERTIES table, which will correspond to the news additional fields. These properties are already known and will be created at runtime:
- Specific properties for the metadata items of OBJECT_TYPE "newsPageVersion", "newsLatestDraftPage" or "newsDraftPage" since they will contain informations related to a draft article, the latest draft article or a specific article version:
- illustrationId: The file id of the article banner, this property value will be specific to the language of the notes draft page, the notes page or the notes page version.
- uploadId: The upload id of the article banner.
- summary: The article summary, this property value will be specific to the language of the notes draft page, the notes page or the notes page version.
- attachments_ids: The jcr uuids of the last article version attachments.
- A specific property for metadata items of OBJECT_TYPE "newsLatestDraftPage":
- activityPosted: A boolean value which indicates if the corresponding generated activity of the lastest draft articleis posted in the space activity stream.
- Specific properties for only metadata items of OBJECT_TYPE "newsPage" since they will contain informations related to the last article version and not those of a draft article, a latest draft article or a specific article version:
- published: A boolean value which indicates if the last article version is published.
- activityPosted: A boolean value which indicates if the corresponding generated activity of the last article version is posted in the space activity stream.
- publicationState: The current publication state of the last article version.
- scheduled: A boolean value which indicates if the last article version is scheduled.
- schedulePostDate: The schedule date of the last article version.
- activities: The shared activities of the last article version activity.
- audience: The publish audience of the last article version.
- viewsCount: The views count of the last article version.
- viewers: The viewers user ids of the last article version.

METADATA_ITEM_PROPERTIES table:

METADATA_ITEM_ID	NAME	VALUE
100	IllustrationId	100
100	summary	My draft article Summary
101	activityPosted	true
101	published	true
101	schedulePostDate	My article version 20 Summary
102	IllustrationId	100
102	summary	My article first version Summary
103	IllustrationId	100
103	summary	My draft for existing article Summary
103	activityPosted	true

.....

azayati commented 5 months ago

I have created this DRAFT Mip with the content discussed during our Meeting of January 23rd. @boubaker Can you take a look, give me your first feedbacks/remarks in order to start discussions ? cc @rdenarie @margondicco @srenault-meeds

plamarque commented 5 months ago

Added a rationale

boubaker commented 5 months ago

IMHO the Metadata API is not the good solution to use here. In fact, we are designing a new Data Model for a new feature which will reuse the notes editor, but not the data model of notes as it is. The Metadata API is designed to decorate mixed types of data, such as adding favorites on spaces, users, notes, activities ... Using it to design a new data model on top of an existing data Model can be used as a trick but not for design a new addon in product.

In addition, polluting the Notes data model with News fields and Business logic can make both products unmaintainable and hard to evolve. I propose to study more the Software Architecture by bringing a second option where we use a specific database Model for News (duplicating code in this special case, can reduce complexity and reduces the cost of flexibility IMO). Eager to discuss more about this.

azayati commented 5 months ago

@boubaker If I have understand your point, you mean that News should have its own tables (exactly like notes ones) + the specific news fields. A duplication of code will be done also for all notes layers + search. In that case IMO, no relation between a news entity and a notes entity, we are no more talking about a migration of news data model to notes one, but it will be an upgrade of news data model from jcr to jpa.

cc @rdenarie @margondicco

boubaker commented 5 months ago

@boubaker If I have understand your point, you mean that News should have its own tables (exactly like notes ones) + the specific news fields. A duplication of code will be done also for all notes layers + search. In that case IMO, no relation between a news entity and a notes entity, we are no more talking about a migration of news data model to notes one, but it will be an upgrade of news data model from jcr to jpa.

cc @rdenarie @margondicco

For my part I think that we have the following disadvantages with current design:

The News table fields are intrusive inside Notes Data Model which shouldn't change to support News, but News has to extend Notes to add more information
Changing Data Model and business logic of notes would add complexity to the current behavior of notes
Changing Data Model and business logic of notes would add instability to an existing addon which is quite stable
Adding an addon business logic inside another addon which doesn't use it, makes the maintainability and evolutivity quite hard to do

To design an addon which will extend another (like we do in all other type of addons), we have to add extension points (using extension registry for the front end And Plugins for the backend). Considering this, IMO, we should rethink the technical design to benefit from Notes existing behaviors instead. There is no functional requirement in the MIP, thus it will be difficult to design a good solution. But if :

we aim to be able to switch a Note to a News
we aim to reuse the Notes versioning system
we aim to add on top of the Notes Page some additional fields when it's switched to a News
we aim to reuse the Notes editor, with additional editable fields when it's a news type

Then, like we do for Activities, we should keep the original behavior of Notes without changing it and just add a Data Model Table to decorate the Notes Object with additional fields like:

Summary
Title
Published Version
Publisher
Publication Date ...

Those information should be stored and managed outside of Notes and managed in UI using extensionRegistry extensibility.

PS: as the current technical requirement doesn't use Metadata API for adding additional fields (like Summary & Title), I think it would be better to not use it for News for now and keep the design of using Metadata Model when a dedicated MIP to CMS is proposed. (It would simplify the development of current MIP, but I let you choose between using a dedicated Table for News or Using the Metadata API)

azayati commented 5 months ago

I vote for using dedicated tables for news and not the metadata, since the data model of news will complete the "Notes" one, we should have:

Three tables like for notes (NEWS, NEWS_VERSIONS, NEWS_DRAFT) having the id of the corresponding note object.
Should we add a type field in "Notes" tables in order to retrieve the stored notes (with type "News") ? or we retrieve them based on the notes ids referenced from the "News" tables ?

boubaker commented 4 months ago

Should we add a type field in "Notes" tables in order to retrieve the stored notes (with type "News") ? or we retrieve them based on the notes ids referenced from the "News" tables ?

I let you decide, from my part not changing notes but making it extensible to support News features through extensionRegistry & plugins is sufficient for me.

Julien-Dubois-eXo commented 3 months ago

Hello @srenault-meeds can you check the functional part please? FYI these elements are those we have to work on during the migration. They will be improved during the next functional iteration.

srenault-meeds commented 3 months ago

Ok for me

azayati commented 2 months ago

@boubaker can you check technical part ?

boubaker commented 2 months ago

@boubaker can you check technical part ?

Ok, what about the content addon software architecture ?

azayati commented 2 months ago

@boubaker can you check technical part ?

Ok, what about the content addon software architecture ?

The content addon is the same as news addon without jcr/ecms dependencies, the storage of news will be based on notes api and metadata api as explained here

boubaker commented 2 months ago

@boubaker can you check technical part ?

Ok, what about the content addon software architecture ?

The content addon is the same as news addon without jcr/ecms dependencies, the storage of news will be based on notes api and metadata api as explained here

Ok, I would prefer to know more about it and how you will use the new Storage layer for News in Meeds Context. But let's keep this first version of technical specification, knowing that it will be difficult to give a go-merge if I wasn't aware about the real technical choices made. In other terms, I will have to make a reverse-engineering of PRs to know how have you dealt with the migration from JCR to the new Storage Layer in News and if it fits the original concern: use a generic storage for CMS Based applications with a first concrete usage on News. Go Tech added.

azayati commented 2 months ago

Ok, I would prefer to know more about it and how you will use the new Storage layer for News in Meeds Context. But let's keep this first version of technical specification, knowing that it will be difficult to give a go-merge if I wasn't aware about the real technical choices made. In other terms, I will have to make a reverse-engineering of PRs to know how have you dealt with the migration from JCR to the new Storage Layer in News and if it fits the original concern: use a generic storage for CMS Based applications with a first concrete usage on News. Go Tech added.

You are right, I will add all details in the description @boubaker

margondicco commented 2 months ago

Hello, after discussions and in order to deliver a smaller scope, we made the following choices:

The following requirement are moved in another MIP :

JCR News articles migration
Scheduled publication
Following filters in news app : posted articles, published articles, my posted articles, archived articles, scheduled articles
Search news articles by keyword and by spaces in news app
Search news articles from unified search
In order to hide the options scheduled publication and filters, a flag is expected
Analytics and gamification impact

So it will be possible with this MIP to :

Post an article in the stream
Publish an article in one or more newslists through a target
Modify or delete an article
Manage drafts in the news app

In addition, will be delivered:

Management of permissions (space with or without redactors) and drafts (share draft mandatory with members or only with redactors FOR notes and news)
Two page models: section and news

cc @Julien-Dubois-eXo @srenault-meeds @azayati

srenault-meeds commented 2 months ago

Ok for me

azayati commented 2 months ago

Ready for Review by DAO Members ( Meeds: @srenault-meeds @boubaker)

PRs and ACCs Up and Running (https://mips-ft.meeds.io/)

boubaker commented 2 months ago

Ready for Review by DAO Members ( eXo : @srenault-meeds @boubaker)

PRs and ACCs Up and Running (https://mips-ft.meeds.io/)

We will wait the Functional validation next week due to unavailability.

srenault-meeds commented 1 month ago

hello @Julien-Dubois-eXo

I have only one question regarding news drafts, as I understand, we have access to news drafts with this MIP but it seems to not work, the page is empty (with redactor access or not)

cc @azayati @margondicco

azayati commented 1 month ago

hello @Julien-Dubois-eXo

I have only one question regarding news drafts, as I understand, we have access to news drafts with this MIP but it seems to not work, the page is empty (with redactor access or not)

cc @azayati @margondicco

@srenault-meeds it is an isolated case of editing an existing article, the corresponding draft wiki page is not stored in the WIKI-DRAFT_PAGES table but stored in SOC_METADATA_ITEMS table which causes the problem. I have tested again the edition of an existing article without any problem. I need the exact scenario to reproduce that.

srenault-meeds commented 1 month ago

I don't reproduce it anymore sorry... so let's go

azayati commented 1 month ago

I don't reproduce it anymore sorry... so let's go

@srenault-meeds we have found the case, edit of an existing article then edit post/publishing of the same article. We will fix that before merging

srenault-meeds commented 1 month ago

Ok thanks. At least, I have not dreamt about it :D

azayati commented 1 month ago

Ok thanks. At least, I have not dreamt about it :D

@srenault-meeds it is fixed, you can validate. @boubaker can you approve PRs ?

boubaker commented 1 month ago

Ok thanks. At least, I have not dreamt about it :D

@srenault-meeds it is fixe, you can validate. @boubaker can you approve PRs ?

Since we changed the MIP process delivery, no need anymore for PRs approval. If you get a Go from @srenault-meeds, it's ok for me to merge as well.

srenault-meeds commented 1 month ago

Ok LGTM. Thanks

Meeds-io / MIPs