[Research] Improve handling of interactive elements in EPUB/PDF exports

[SteelWagstaff]

When a book includes an audio, video, iframe, or H5P element, we insert a fallback placeholder informing the reader that this element has been excluded from the EPUB or PDF export and providing a link to the element in the book itself. See https://github.com/pressbooks/pressbooks/tree/3217c734b9f3d25d840c247a5ac5b16b6c1698cb/templates/interactive

I'd like us to research the feasibility of the following:

1. When the excluded activity is an H5P, can we print the full activity itself and let the user know that "This is a static version of an interactive H5P activity. To see the full interactive element, visit chapterlink#h5pAnchor". See https://github.com/pressbooks/pressbooks/issues/2189 for similar idea.
2. When there are multiple audio, video, or iframe elements inserted consecutively (without non-media content between them), can we simply print a single placeholder that reads: "Multiple audio, video, or iframe elements have been excluded from this version of the text. You can find them online here: 'link to first of the excluded elements' (ideally with anchor)." instead of adding each of them separately? See screenshot below for an example of what we want to avoid:
3. Can we insert and link to anchors for all other excluded activities like we do now for H5P activities? https://github.com/pressbooks/pressbooks/pull/2153

[richard015ar]

I was understanding how the media content is processed in our XHTML export routine. I was trying to book a meeting with Os at the end of the sprint, but we could not arrange a time, since both were with many meeting. This was that I debugged and learned about our media content process:

• We process the the HTML data before export using the tidy XHTML class method.
• This method replaces the interactive tags using the Content class
• We replace video and audio tags calling to the templates referencing to: https://github.com/pressbooks/pressbooks/tree/3217c734b9f3d25d840c247a5ac5b16b6c1698cb/templates/interactive
• In that step is where we can send the links to with the excluded activities (point 3 in the ticket's description): https://github.com/pressbooks/pressbooks/blob/7ea8dd2aff224cfdf1bab8d119f1588e44772a72/inc/interactive/class-content.php#L315-L318

[richard015ar]

Google PageSpeed provides a support for screenshots on server side. Testing , for example: https://pagespeedonline.googleapis.com/pagespeedonline/v5/runPagespeed?url=https://integrations.pressbooks.network/ltidemo/chapter/licao-1/#h5p-1&category=CATEGORY_UNSPECIFIED&strategy=DESKTOP&key= It takes too much time to process and as expected it does not work locally (since the URLs generated are local).

[SteelWagstaff]

For #1, can we try to render the content of the HTML activity in the XHTML file before producing the PDF? Look at H5P shortcode generate pattern in their plugin. If this works, render this instead of the current H5P interactive content fallback with this message: "This is a static version of an interactive H5P activity. To see the full interactive element, visit chapterlink#h5pAnchor"

For #2, can we try a CSS rule in EPUB and PDF exports that looks something like .interactive-content + .interactive-content { display: none; }. We can decide about a modified message later.

For #3, I'd like us to add anchor links for all interactive content that we replace in the EPUB/PDF exports, just like we do with H5P activities

[richard015ar]

For point 2: using CSS rule suggested in PDF and EPUB it looks like:

Instead:

It seems to work as expected!

[SteelWagstaff]

@richard015ar The PR you submitted works as expected for audio and video elements, but doesn't add anchor links for oEmbed elements or other iframed content. Is it possible to add anchors for these other elements as well?

[richard015ar]

What I got so far:

• Anchors links added for audio, video, oembed and h5p. There is a PR to be reviewed: https://github.com/pressbooks/pressbooks/pull/2508

• Some interactive elements rendered using blade generates extra break lines. I created a method that right after add the interactive contents, remove the unwanted break lines and worked in order to apply the CSS rule to group the interactive contents. This is the method that worked:

     /**
   * Remove breaklines after interactive contents elements rendered.
   *
   * @param $dom
   * @return mixed
   */
  public function removeBreaklinesBetweenInteractiveContents( $dom ) {
      $nodes = $dom->getElementsByTagName( 'div' );
      for ( $i = 0; $i < $nodes->length; $i++ ) {
          $div_element = $nodes->item( $i );
          if ( strpos( $div_element->getAttribute( 'class' ), 'interactive-content' ) !== false ) {
              $interactive_element = $div_element;
              $next_element = $interactive_element->nextSibling;
              if (
                  ! is_null( $next_element ) &&
                  ! is_null( $next_element->nextSibling ) &&
                  $next_element->nextSibling->nodeName === 'br'
              ) {
                  $next_element->nextSibling->parentNode->removeChild( $next_element->nextSibling );
  
              }
          }
      }
      return $dom;
  }

  The problem with this is we inspecting all the divs in each chapter. Probably, instead remove the break lines, we could just remove the interactive contents, which would require a recursive method to inspect if next to the current interactive content element, exists a next one, etc.

• The CSS rule was added here. Probably it needs to be added in other themes as well. Anyway I didn't submit a PR for this, since the break line problem needs to be solved, and I spent enough time trying to create a workaround with that.
• Other problem is h5p activities does not contains the .interactive-content class, which it is grouped separately. Probably it is something desired.
• The iframes and oembeds are wrapped in <p> tags, except when those are consecutive. Which means, in order to group those using a CSS rule, we would need to restructure the way we render those. I was able to see where some specifics iframe (specific providers) are rendered, and the oembeds. Nevertheless, when the user add a custom iframe, looks like are not using any custom renderization. We would need to hook those.

Next step in this task is research the h5p activities renderization for exports.

[richard015ar]

For h5p render in exports

• We can get raw data from H5P using the plugin just like we do in some places. For an sample activity we would get something like:

  {
  "presentation": {
  "slides": [
    {
      "elements": [
        {
          "x": 29.956427015250547,
          "y": 29.999999999999996,
          "width": 40,
          "height": 40,
          "action": {
            "library": "H5P.AdvancedText 1.1",
            "params": {
              "text": "<p>i am a text</p>\n"
            },
            "subContentId": "135cf6de-6bf1-4ae2-8546-07b3d4fcb337",
            "metadata": {
              "contentType": "Text",
              "license": "U",
              "title": "Untitled Text"
            }
          },
          "alwaysDisplayComments": true,
          "backgroundOpacity": 0,
          "displayAsButton": true,
          "buttonSize": "big",
          "goToSlideType": "specified",
          "invisible": false,
          "solution": "<p>comments here</p>\n"
        },
        {
          "x": 50,
          "y": 24.946236559139784,
          "width": 27.067787594503056,
          "height": 40,
          "action": {
            "library": "H5P.Image 1.1",
            "params": {
              "contentName": "Image",
              "file": {
                "path": "images/file-619284f6326bd.jpg",
                "mime": "image/jpeg",
                "copyright": {
                  "license": "U"
                },
                "width": 1600,
                "height": 1197
              },
              "alt": "alt"
            },
            "subContentId": "6faf86ed-84b5-4646-b56e-bd79b6ad5c15",
            "metadata": {
              "contentType": "Image",
              "license": "U",
              "title": "Untitled Image"
            }
          },
          "alwaysDisplayComments": false,
          "backgroundOpacity": 0,
          "displayAsButton": false,
          "buttonSize": "big",
          "goToSlideType": "specified",
          "invisible": false,
          "solution": ""
        }
      ],
      "slideBackgroundSelector": {}
    }
  ],
  "keywordListEnabled": true,
  "globalBackgroundSelector": {},
  "keywordListAlwaysShow": false,
  "keywordListAutoHide": false,
  "keywordListOpacity": 90
  },
  "override": {
  "activeSurface": false,
  "hideSummarySlide": false,
  "summarySlideSolutionButton": true,
  "summarySlideRetryButton": true,
  "enablePrintButton": false,
  "social": {
    "showFacebookShare": false,
    "facebookShare": {
      "url": "@currentpageurl",
      "quote": "I scored @score out of @maxScore on a task at @currentpageurl."
    },
    "showTwitterShare": false,
    "twitterShare": {
      "statement": "I scored @score out of @maxScore on a task at @currentpageurl.",
      "url": "@currentpageurl",
      "hashtags": "h5p, course"
    },
    "showGoogleShare": false,
    "googleShareUrl": "@currentpageurl"
  }
  },
  "l10n": {
  "slide": "Slide",
  "score": "Score",
  "yourScore": "Your Score",
  "maxScore": "Max Score",
  "total": "Total",
  "totalScore": "Total Score",
  "showSolutions": "Show solutions",
  "retry": "Retry",
  "exportAnswers": "Export text",
  "hideKeywords": "Hide sidebar navigation menu",
  "showKeywords": "Show sidebar navigation menu",
  "fullscreen": "Fullscreen",
  "exitFullscreen": "Exit fullscreen",
  "prevSlide": "Previous slide",
  "nextSlide": "Next slide",
  "currentSlide": "Current slide",
  "lastSlide": "Last slide",
  "solutionModeTitle": "Exit solution mode",
  "solutionModeText": "Solution Mode",
  "summaryMultipleTaskText": "Multiple tasks",
  "scoreMessage": "You achieved:",
  "shareFacebook": "Share on Facebook",
  "shareTwitter": "Share on Twitter",
  "shareGoogle": "Share on Google+",
  "summary": "Summary",
  "solutionsButtonTitle": "Show comments",
  "printTitle": "Print",
  "printIngress": "How would you like to print this presentation?",
  "printAllSlides": "Print all slides",
  "printCurrentSlide": "Print current slide",
  "noTitle": "No title",
  "accessibilitySlideNavigationExplanation": "Use left and right arrow to change slide in that direction whenever canvas is selected.",
  "accessibilityCanvasLabel": "Presentation canvas. Use left and right arrow to move between slides.",
  "containsNotCompleted": "@slideName contains not completed interaction",
  "containsCompleted": "@slideName contains completed interaction",
  "slideCount": "Slide @index of @total",
  "containsOnlyCorrect": "@slideName only has correct answers",
  "containsIncorrectAnswers": "@slideName has incorrect answers",
  "shareResult": "Share Result",
  "accessibilityTotalScore": "You got @score of @maxScore points in total",
  "accessibilityEnteredFullscreen": "Entered fullscreen",
  "accessibilityExitedFullscreen": "Exited fullscreen"
  }
  }

  Then we would use some h5p script to render it properly, which is the way the plugin does. We could include one or more (on demand) of the following scripts: https://github.com/h5p/h5p-php-library/tree/1a644dda1134b1ce60635db8d0ab5f09caaa03e4/js

Next step here would be: understand how the scripts works to render the content and use those in our export routine.

Unfortunately h5p activity does not provide a on demand solution to render the content, and I suspect the main reason is because most of the content use a ton of javascript, which are not only necessary for the interaction, but also for rendering the content.

[richard015ar]

I left a question to the H5P people about the possibility to get a preview to export it: https://h5p.org/documentation/setup/wordpress?page=2#comment-44029

[fdalcin]

Tested pressbooks/pressbooks#2508 on integrations and it works as expected, see below:

EPUB

PDF

PDF version has the long URL that we discussed.

[SteelWagstaff]

Findings:

1. We don't have a good native H5P render method -- producing these in our XHTML export would be very difficult
2. We switched to using a shared template for most interactive elements and added anchor links to all interactive fallback elements EXCEPT custom iframes. In order to do it for these, we would need to inspect post content and hook into these. Doable, but not done yet.
3. We did not implement a method for hiding consecutive interactive elements. There are two possible approaches, both of which require analysis of the whole DOM for each chapter. Method 1 is simpler but less performant (analyze DOM and remove breaklines between interactive elements generated by Blade and then apply CSS to hide those that follow immediately after others) and Method 2 is to analyze the DOM and simply remove activities that follow others which would require a bigger refactor than method 1. A third, simpler method would be to make sure that the blade templates never produce empty paragraphs after them and then write CSS like: .interactive-content + br + .interactive-content, .interactive-content + .interactive-content { display: none; }