Epic: GREI - Dataverse Rearchitecture

cmbz commented 10 months ago

Overview

"Improve three areas of the repository UI: 1) facilitate discovery and exploration of data by making it easier to customize collections of datasets, identify types of data more clearly, expand and improve tools to explore data within repositories, 2) expand and clarify the dataset creation and publishing workflow to make it easier to add a wider variety of data with appropriate metadata, 3) extend the RESTful APIs in the Dataverse software to enable more easily separating the backend web services that support the core Dataverse functionality with the frontend, and build scalable and flexible open-source UI modules for the Harvard Dataverse repository, usable by other repositories." (Was NIH OTA Aim 7)

Try out SPA Beta here: https://beta.dataverse.org/spa

Tasks

2024 Q2

See: Proposal: SPA Beta Features for Q2 2024
Note: Epic completeness varies according to year, quarter, and goals
Infinite Scrolling
Creation of datasets
Editing datasets metadata
Basic file upload
Basic creation of collections (nice to have)
Publishing datasets (nice to have and candidate for community contribution)

2024 Q3

See proposal: SPA Beta Features for Q3 2024
Q3 planning in progress
Collection page results of all types (Files, Datasets and Collections)
Collection page basic filtering options and search
Search home page (from the epic)
Create Collection page (Full version)
Enhanced File upload

2024 Q4

Pending

Resources

https://github.com/IQSS/dataverse-pm/issues/124
Also, see the following project boards:
- Global Backlog: Resarch SPA Beta and Rearch Auth
- Sprint Board
- Project Google Folder (includes weekly status reports)

cmbz commented 9 months ago

Status: January 2024 Update

Work continues towards accomplishing the Must Have SPA features (described in: Proposal: SPA Beta Features) in advance of the SPA Beta demo at the upcoming Dataverse Community Meeting, 2024 and the 6.2 release, scheduled for 2024/03/31. Progress includes a functional Home Page, File Page, and elements of the Dataset Creation Page. Several backend API extensions to support the SPA were also completed.

Completed

cmbz commented 8 months ago

Status: February 2024 Update

Work continues towards accomplishing the Must Have SPA features (described in: Proposal: SPA Beta Features) in advance of the SPA Beta demo at the upcoming Dataverse Community Meeting, 2024 in March, and the 6.2 release, scheduled for 2024/03/31. Progress includes: must-have improvements to the Home Page, File Page, and page navigation, as well nice-to-have improvements to the Dataset Creation Page. In addition, the team is developing a new SPA proposal for Q2 which will be made available for community comment.

Specific completed issues include:

Completed

dataverse

dataverse-javascript-client

dataverse-frontend

cmbz commented 8 months ago

Status: March 2024 Update

A demo of the new SPA design and implementation (Dataverse SPA Development) was presented at the Dataverse Community Meeting, 2024 in Texcoco, Mexico.
Work continues towards next SPA milestones.

Completed

cmbz commented 7 months ago

Status: April 2024

Substantive progress has been made on the Collections and File Upload pages, improvements to pagination, and documentation. See below for specific PRs merged during April 2024.

Completed

dataverse repo

dataverse-frontend

dataverse-client-javascript

cmbz commented 5 months ago

Status: May 2024

See details here: Rearchitecture Front End Weekly Update 🔁 - May 30th

Several features are in progress, are in Q/A, or have been merged:

Host Collection field to the Create Dataset Form (Merged)
Multiple dynamic fields (Merged)
Upload Files Page in progress
Edit Dataset (next big Q2 feature) will re-use code from Create Dataset
Edit Dataset use case has been developed and merged
Create Dataset page: Match API Email validation (Q/A)
Extend Breadcrumbs generator for action items (Merged)
Collection page: Bug addressed for collection page datasets not being refetched after navigating from a sub collection (Ready for QA)
Trigger github actions on PR (Merged)

Completed

dataverse

dataverse-frontend

dataverse-client-javascript

https://github.com/IQSS/dataverse-client-javascript/pull/151

cmbz commented 5 months ago

Status: June 2024

This month, the SPA saw the following accomplishments and progress:

File Upload
- File Upload use cases (Merged) https://github.com/IQSS/dataverse-client-javascript/pull/153
- File Upload UI (Merged) https://github.com/IQSS/dataverse-frontend/pull/397
- File Upload use cases (Ready for QA) https://github.com/IQSS/dataverse-client-javascript/pull/153
- File Upload Integration (Work In Progress): https://github.com/IQSS/dataverse-frontend/issues/417
Dataset page
- Infinite scrolling on dataset files (Merged) https://github.com/IQSS/dataverse-frontend/pull/415
- Storybook example
Edit Dataset metadata page
- https://github.com/IQSS/dataverse-frontend/pull/422 (Ready for Review)
Create collection page
- Basic use case for creating a collection (Ready for QA) https://github.com/IQSS/dataverse-client-javascript/pull/155
Fix dataverses/{id}/metadatablocks endpoint
- (In Review) https://github.com/IQSS/dataverse/pull/10642
Create Dataset page
- Prefill form fields with user info (Merged) https://github.com/IQSS/dataverse-frontend/pull/414
- Match API Email validation (Merged) https://github.com/IQSS/dataverse-frontend/pull/402
Q3 feature proposal doc ready for review
- https://docs.google.com/document/d/1n13bBaYDpbs8TouXKlZfX07v2D4ZVfjabip4vu-kjEQ

cmbz commented 4 months ago

Status: Year 3, Q1

Completion: Pending (updated for Q2)
Year 3 Q1 saw a number of SPA developments, including a demo of the new SPA design and implementation (Dataverse SPA Development) presented at the Dataverse Community Meeting, 2024 in Texcoco, Mexico. Substantive progress was made on the Collections and File Upload pages, such as improvements to pagination and documentation and infinite scrolling support. Additional accomplishments included enhancing the Create Dataset Form, support for multiple dynamic fields, and finalizing the Edit Dataset use case.
See: NIH GREI Quarterly Progress Report: Year3 Q1 (March 2024 - May 2024)

cmbz commented 3 months ago

Status: July 2024

Update

Proposal: SPA Beta Features for Q3 2024 was completed
Project was rebaselined to establish completeness.
- Completeness: 30%

Features and Functionality

Detailed update: Rearchitecture Frontend Weekly Update (2024-08-01)

Dataset page

Fix dataset files scroll selection
- https://github.com/IQSS/dataverse-frontend/pull/430 (Merged)
Support Update Current Version on Publish Dataset use case
- https://github.com/IQSS/dataverse-client-javascript/pull/166 (Merged)
Fix Dataset view metadata missing fields
- https://github.com/IQSS/dataverse-frontend/pull/443 (Merged)

Create Dataset and Edit Dataset metadata page

Fix date format validation on create & update dataset JS-Dataverse
- https://github.com/IQSS/dataverse-client-javascript/pull/161 (Merged)
Select Advanced for Design System to select one or multiple options.
- https://github.com/IQSS/dataverse-frontend/pull/424 (Merged)

Create Collection page

Extension of the create collection use case to support a full page version.
- https://github.com/IQSS/dataverse-client-javascript/pull/162 (Merged)
Extension of the add dataverse API endpoint to support the use case.
- https://github.com/IQSS/dataverse/pull/10644 (Merged)

List Completed Issues & Pull Requests

Issues

Pull Requests

cmbz commented 2 months ago

Status: August 2024

Update

Completeness: 32.83%
Features and Functionality
Detailed update: Rearchitecture Front End Weekly Update (2024-08-29)

Collection Page

CollectionCard Component
- https://github.com/IQSS/dataverse-frontend/pull/459 (Merged)
Adding publication status to search results
- https://github.com/IQSS/dataverse/pull/10738 (Merged)
FileCard Component
- https://github.com/IQSS/dataverse-frontend/pull/450 (Merged)
Collection page URL
- https://github.com/IQSS/dataverse-frontend/pull/453 (Merged)

Create Collection Page

Transfer List Component
- https://github.com/IQSS/dataverse-frontend/pull/458 (Merged)
Fixing API issue related to displayOnCreate property and inputLevels
- https://github.com/IQSS/dataverse/issues/10741 (Merged)
Adds new use case GetAllMetadataBlocks
- https://github.com/IQSS/dataverse-client-javascript/pull/172 (Merged)
Adds GetCollectionUserPermissions use case
- https://github.com/IQSS/dataverse-client-javascript/pull/175 (Merged)
Metadata fields section on Create Collection page
- https://github.com/IQSS/dataverse-frontend/pull/451 (Merged)

File Uploading

Enhanced file uploading
- https://github.com/IQSS/dataverse-client-javascript/pull/173 (Merged)
Configure s3 in the containerized environment for file upload e2e tests
- https://github.com/IQSS/dataverse-frontend/pull/457 (Merged)

Dataset Page

Fix dataset files scroll selection
- https://github.com/IQSS/dataverse-frontend/pull/430 (Merged)
Support Update Current Version on Publish Dataset use case
- https://github.com/IQSS/dataverse-client-javascript/pull/166 (Merged)
Fix Dataset view metadata missing fields
- https://github.com/IQSS/dataverse-frontend/pull/443 (Merged)

Create Dataset and Edit Dataset metadata Page

Fix date format validation on create & update dataset JS-Dataverse
- https://github.com/IQSS/dataverse-client-javascript/pull/161 (Merged)
Select Advanced for Design System to select one or multiple options.
- https://github.com/IQSS/dataverse-frontend/pull/424 (Merged)

Create collection Page

Extension of the create collection use case to support a full page version.
- https://github.com/IQSS/dataverse-client-javascript/pull/162 (Merged)
Extension of the add dataverse API endpoint to support the use case.
- https://github.com/IQSS/dataverse/pull/10644 (Merged)

cmbz commented 2 months ago

Status: Year 3, Q2

Completion: 32.83%
The frontend team continued to implement the functionality described in the Proposal: SPA Beta Features for Q3 2024 (note that the SPA year begins in January).
Progress was made towards completing UI elements, including the Collection Page, Create Collection Page, Dataset Page, and the Create Dataset and Edit Dataset Metadata Page. File uploading was also enhanced, and improvements were made to the API to support email validation.

cmbz commented 2 months ago

Status: September 2024

Highlights

Account page:

API token management use cases
- https://github.com/IQSS/dataverse-client-javascript/pull/193 (Ready for QA)
- Includes new token's expiration date in "users/token/recreate" endpoint. See https://github.com/IQSS/dataverse/pull/10858 (QA)

Multiple pages:

Private route guards
- https://github.com/IQSS/dataverse-frontend/pull/483 (Merged)
Fix end to end test
- https://github.com/IQSS/dataverse-frontend/pull/502 (Ready for QA)

Collection page:

Publish Collection
- https://github.com/IQSS/dataverse-frontend/pull/500 (Ready for Review)

cmbz commented 1 month ago

Status: October 2024

Complete: 35.84%

Highlights

Q4 features proposal

https://docs.google.com/document/d/1Z9IHtjmbvWpmybk8ObifvfoF1IAGCZiv6ZNz6MamKFQ

Account page API token management use cases

https://github.com/IQSS/dataverse-client-javascript/pull/193 (Merged)
Includes new token's expiration date in "users/token/recreate" endpoint
- https://github.com/IQSS/dataverse/pull/10858 (Merged)

Collection page

Show results of all types + search & filters

https://github.com/IQSS/dataverse-frontend/pull/509 (Merged)

Publish Collection

https://github.com/IQSS/dataverse-frontend/pull/500 (QA)

Add isReleased to collection DvObjectOwnerNode

https://github.com/IQSS/dataverse-client-javascript/pull/199 (QA)

Dataset page

Publish Dataset and Parent Collection

https://github.com/IQSS/dataverse-frontend/pull/520 (In Review)

General

Implement React Lazy imports (lazy loading)

https://github.com/IQSS/dataverse-frontend/pull/512 (Ready for QA)

Add error handling UI page

https://github.com/IQSS/dataverse-frontend/pull/518 (In Review)

E2E fix

https://github.com/IQSS/dataverse-frontend/pull/502 (Merged)

https://github.com/IQSS/dataverse-frontend/pull/502 (Merged)

IQSS / dataverse-pm