Pagination: ACSS

[dragontheory]

Show/hide based on dividing total number of visible search result rows with total search results returned.

Note: search result rows will have options to view as cards and will be responsive. In that case, we may need to go to an infinite scroll option via CSS media queries.

[dragontheory]

Another reason to get as much out there as possible for the end of the month. That much less for them to wait on me for.

[bob2517]

Cool :) Almost there - just had to adjust the API so it was doing things properly. I just got the superhero names popping up in the list, so will do the other bits now.

[dragontheory]

Please forgive my ignorance but I want to better understand. You're language of choice is PHP. Is that the language you are using for the back-end? Or is that more of a middleware language?

[dragontheory]

Many of the guys I work with use Python for the back-end language.

[dragontheory]

Getting late for you. Better catch some sleep.

[bob2517]

lol!

I'm using PHP for the back-end. It doesn't really matter what language is on the back-end - they all do the same thing. I'm just using PHP because I'm the most familiar with it.

PHP is a raw language - the nginx web server runs the program directly in the PHP language, so I guess it's just like Python. Python is trendier than PHP, but PHP is fine too and very fast, if you are not using middleware with it.

Most devs use PHP with middleware - like Laravel - and I do that at the workplace to stay up with the modern trends - most of the time. But for this I just went raw PHP, as it's quick. It doesn't matter in terms of industry standards - a JSON response can be provided in any back-end language. Whatever is scalable and fast and what your devs are familiar with is the solution.

In terms of data, like superheroes - I wouldn't worry about it particularly. Data is just data. But if you do want to change the API, let me know in the coming week and give me a JSON file or something.

Yes - I'm on a roll - will be done shortly...

[dragontheory]

All good. Just getting it straight in my head. lol

Data is just data. But if you do want to change the API, let me know in the coming week and give me a JSON file or something.

Meaning, if I want to use something other than the superhero JSON?

[bob2517]

Yeah, exactly. I've got the code set up now, it would be a case of re-adjusting the code to use the different fields.

I've just done a commit to the branch, but it's not done yet. There's an issue with the number of table cells which I need to look at. Will do that tomorrow, as I'm beginning to wane.

I need a boost of Luke Cage.

G'night and will do more tomorrow.

[dragontheory]

lol. Nice. Thanks for all that you did today.

Have a good evening.

[bob2517]

I just put up a basic working copy with the supe data on the branch. The API is noticeably faster (for me anyway) than the names API. Let me know if it's faster for you. My server is in London, but it shouldn't make a huge amount of difference for you. The benefits of running an 8GB SSD dedicated virtual server for the docs site...

I had a thought last night about having a configuration option at the top of the ACSS, which would allow the developer to specify field name and field headings. That way they wouldn't have to edit the code itself to add a new data schema. The schema could even be stored in a separate JSON file and loaded up prior to drawing content, or even embedded in a XML type tag somewhere on the page. Let me know if you want me to set that up - it's a relatively quick thing to do. I've used that method before on a couple of back-end frameworks that I've built and it's a good stable way of separating functionality from data. It would also mean that I wouldn't have to muck about with the fields to get it to work when there is a change :)

The ajax call addresses for row and aside should also be extracted out from the code and put into variables for use in the command.

I think having them set as variables makes more sense and keeps them all together without any unnecessary HTML markup - they could be declared at the point of preInit.

That way if the data schema changes, the changes to the code only have to happen in one place.

You know what? I'm just going to do it and show you. We can always revert it later on if needed.

[dragontheory]

That way if the data schema changes, the changes to the code only have to happen in one place.

Good morning! Happy Lord's Day!

Yeah. That sounds good. Thank you!

[dragontheory]

You mentioned something a while ago about built in CORS mitigation. Is that something that was implemented? No worries if it's not. Just a CYA moment. Envisioning myself talking about it to other DEVs and wanted to know if it's actually there or not.

[bob2517]

Good morning! Ok cool.

CORS - yes, it's possible to add headers to the ajax command now, which allows for CORS validating on the server.

The docs for that are in the ajax command.

[bob2517]

Got the first bit of that working - I had to change the way attributes are handled in the core for it to work though.

The new behaviour when setting up variables, is that if the attribute is not available, then it keeps the attribute reference in place until the variable is used elsewhere and is available. It needs a bit more testing before I commit to the official ACSS branch, but it will be on the project. Previously, if the variable wasn't available, it would substitute in an empty string. It still does that, but not when setting up variables. It keeps the attribute reference intact until the variable is used.

This is what's working so far:

    body:preInit {
        $rowPopulationURL: "https://activecss.org/get-supes.php";
        $asidePopulationURL: "https://activecss.org/get-supe.php?id={@data-id}";
    }

And then later, the ajax commands for the row and the list-item become:

            ajax: "{$rowPopulationURL}" get json await;

/* and */

        ajax: "{$asidePopulationURL}" get json cache await;

[bob2517]

The idea is that the developer only has to edit the preInit event to set things up for their project. Following your specification of keeping the data separate from the logic, then this would appear to be the way to go to keep things simple.

Will sort out the other bits now - then if/when you change your mind about the superhero API, it will only need to be changed in one place in the project.

[bob2517]

It may change a bit though from the examples above - I'm still working it out.

[dragontheory]

then if/when you change your mind about the superhero API, it will only need to be changed in one place in the project.

Ha! Spoken like a true experienced DEV who knows nothing is ever as it seems at first. There are always hidden assumptions and thus there will always be changes...

Most of my thinking is centered around mitigating the possibilities...

Love it.

[bob2517]

Yeah :) Until it's actually working, who knows what it's going to look like? It's going ok so far - almost cracked the row display.

[bob2517]

I've just put a commit on the branch that allows basic API schema definition - so it should work with most schemas, and if it doesn't then it wouldn't be a big deal to upgrade that bit of code. I've designed it so that there is room for expansion if/when the time comes.

In theory the developer should only need to edit the top part of the ACSS. There is more to do on it once you've designed the form fields how you want them. Don't worry if it looks too scary, but do ask if you have any questions on it.

This wouldn't have been very easy to do unless I'd done the recent variable upgrade, so it's a good job that I had it wrapped up before the weekend.

There are two helper functions used in the place where the row and the aside is drawn - escapeHTML and getVar.

escapeHTML() just makes sure that any entities in user content is converted to HTML entities before rendering.

getVar() I added to the core today, and it uses the Lodash _get function which allows object variables to be evaluated based on strings, which is needed for specifying where the field is in the API structure, which could be anywhere. eg. I need to fetch the API variable $item.biography.age, but I have "biography.age" in my API structure, and $item["biography.age"] won't find the right thing because of the way variables work, and it needs to be $item.biography.age. So I use getVar($item, "biography.age") to get the right variable. Do we really need it? The answer is yes but only if APIs have weird structures, and they usually do. https://www.tutorialspoint.com/lodash/lodash_get.htm

[bob2517]

In terms of you doing stuff - ie. the fields for CRUD - do it on the main branch, let me know what you've done, I'll then port it over to the branch and then merge the branch into the master when you have got your head around how the new branch looks.

I will remove the tabs at some point and put them back to spaces for you. I just find it difficult to see what's going on unless I have tabs there.

It's going to be easier for you to do your thing with the fields if you use your original HTML. I can then adjust it to work with the new API handling strategy.

Does that make sense? I think that's probably the easiest for you as you are most familiar with your own code.

Meanwhile I'll crack on with the pagination. I managed to put it off for a day, but I don't have any excuses now.

[dragontheory]

Does that make sense? I think that's probably the easiest for you as you are most familiar with your own code.

Ok. Sounds good.

[dragontheory]

This is great for documentation later.

[bob2517]

Cool. Do yell if something doesn't make sense. I will expand the docs in the code to explain the things that you are having any trouble with, or feel free to add your own notes.

[bob2517]

Due to the nature of the project, it does sometimes have to get a little bit hard-core in technique, but I'll try and keep it to a minimum.

[bob2517]

Heya. What's the purpose of app-panel-scroll?

It's over the top of the last record in the list and throwing out the visible row count, which doesn't account for layers on top of elements. and I'm just wondering if it should be in a different panel rather than on top of the list, but I'm not sure what it does. I can do a workaround for it, but it's bit odd for it to be there. I'll do a workaround for now.

[dragontheory]

Heya. What's the purpose of app-panel-scroll?

I had to add an extra layer of HTML infrastructure to handle the scenario of test multiple sections within a panel

The role of app-panel-scroll is simply to group that section and be able to scroll them, given the space left by other sections.

As of right now, the two elements that are allowed to scroll in the entire app are <panel-list> and <app-panel-scroll>. I will probably make them more generic and combine them into one since their purpose is identical.

As far as pagination is concerned, the only element that should be :observed is , and that, only in the center panel.

So the only selector is

app-panel:nth-of-type(3) panel-section:nth-of-type(1) panel-list {}

Is that helpful?

[dragontheory]

About to check in those HTML structural changes.

[bob2517]

Yeah - makes sense - for some reason I thought it was inside the row panel, but it isn't.

I've taken off the scroll bar (from ACSS - I've not touched the CSS), for the row list. Otherwise we don't need pagination based on visible rows at all, we may as well have a fixed number of rows plus a scrollbar, like there is at the moment.

Do you want a scrollbar on the row panel? If so, pagination will be a fixed number per page. If not, I'll continue with doing the pagination based on visible rows. It is going to get weird though - there could be 100 pages when there is only 1 or 2 records per page, which I can deal with. But I just want to make sure you really want to go that route. It's a little bit like you're asking for a haircut that isn't usually done, and I'm the hairdresser. It's fine if so.

[bob2517]

An alternative could be to have a minimum height on the search area when there are less than 5 records, or something like that. That would force the user to not minimise the window too much.

But you're still going to deal with mobile. Is it easier for the user to scroll, or is it easier for them to hit the page numbers to fetch records they cannot see?

[bob2517]

Scrolling allows for infinite scrolling then too.

[bob2517]

Infinite scrolling is quite easy to do. I can just detect the visibility of the last row and call the next page with ajax, and append the new records to the list.

[dragontheory]

CSS allows for natively driven overflow and scroll behavior with sticky header and footers. Meaning no JS.

No need to get rid of scrollbars. There should never be a full-page scrollbar. If there is, let me know. That means, I've messed something up, which will be more and more likely as the layout gets more and more complicated. The only elements that scroll are <panel-list> and <app-panel-scroll>.

Users will have options on how they want to see their data. List or cards as well as pagination or infinite scroll.

So I think detecting the number of visible rows in that one <panel-list> is all we need to drive the pagination numbers and infinite scroll.

app-panel:nth-of-type(3) panel-section:nth-of-type(1) panel-list {}

[bob2517]

No need to get rid of scrollbars. There should never be a full-page scrollbar.

I'm just talking about the scrollbar that is currently on the row list, and trying to tie that into how the paging is supposed to work if the paging is based on visible records - it's not making any sense to me.

Could you tell me what happens when you scroll down the rows? Does the page number change to page 2 automatically? I'm trying to reconcile the concept of page numbers with number of visible rows plus a scrollbar. I'm sure you've got it straight in your head, but I'm having trouble working out the exact flow of things.

[dragontheory]

Sure. I will try...

First, users are able to toggle either pagination or infinite scroll.

If it's pagination, the total number of rows on the page are divided into the total number of search results and that is how many pagination links we will have at the top of the center panel. There are no scrollbars because there are never any more rows in the <panel-list> element that would cause overflow behavior. There will have to be provisions if there are more than five links, but that is an enhancement for later.

If the users change the filters on the left panel, or resizes the page, or change from rows (or table) view to cards view, and or otherwise change the number of visible rows, then the pagination links automagically update accordingly.

If it's infinite scroll, there are no pagination links at the top of the center panel.

As far as I understand infinite scroll behavior, initially twice the number visible rows are delivered to the DOM. When users scroll down (the only direction they can go) into the second group (they were seeing the first group by default) another group of rows (the number of which match the number of visible rows) are appended to the second group of rows and the first (initial) group of rows are removed from the DOM.

Now that users can go either direction, groups of rows are added/removed depending on which direction they are scrolling.

Did similar things in Flash back in the day with carousels.

This ensures users will never get a loader animation and have to wait (network connectivity not withstanding...) for the next group of rows to load as they scroll and there will never be too many rows to negatively affect performance.

I believe this should also work on mobile devices, because the same element is being :observed for everything, no matter what the size.

Does that make sense?

WDYT?

[bob2517]

That is exactly what I needed, thank you. That was really well explained and it has totally cleared up my confusion. All systems go!

The only thing I would remove is the necessity to remove the first set of records from the DOM when scrolling down. Bear in mind that each set of DOM pages is going to be around 1kb, I don't think it's worth removing these. It should have zero impact on performance in ACSS, because row event listeners are not attached to each element, so the overhead is at its minimum in terms of events. Additionally people tend to scroll up faster than scrolling down, so there would be a lag if you have to re-fetch data. Unless the user is scrolling 100s of thousands of pages in one sitting, which is unlikely, filling up memory with megabytes of data, keeping the rows present should never be an issue. 'Tis the beauty of the way ACSS handles events.

Also, each new page will wipe the JSON row strings from memory, as long as the data isn't cached, so there would be no memory leak with regards row data.

If my analysis of that is wrong and there is a performance issue, I will remove the earlier pages, but I don't think it's needed based on earlier experience with infinite scroll.

I can see why you would do it with images in a carousel though - the memory does quickly build up. There are usually no images in row data though, and in my experience images in rows are usually limited to icons or very small re-usable images.

[dragontheory]

OK cool.

Sure. If you don't think we need to remove them. That sounds good.

The only reason I brought it up is because five or so years go, a fellow (back-end) DEV and I tested to see how many rows would have to be loaded into the DOM before there was significant slow down or jankiness (technical term) in scrolling. At the time, it was around 300, depending on the browser, type and amount of data per row.

There were some users who wanted to see ALL the data at once but we had to put at limit of up to 300 rows. As you say, these were extreme and rare cases, but they did exist and we had to code against them.

I agree with you. If a user has that many records at once, they are doing search wrong. They have to use a filter.

[dragontheory]

I failed to mention that, as a general overall strategy, I would like to prestage as much of the UI as possible with CSS doing as much of the DOM heavy lifting as possible. I don't want you to have to write custom code.

This...

1. (hopefully) means less coding for you,
2. helps to reduce the app's dependency on ACSS,
3. helps me and my dependency on you and not having to bother you all the time,
4. and allows me flexibility/freedom to change things if need be (most likely) without necessarily changing the ACSS.

Ideally I would be able to change CSS logic and not have to change underlying ACSS logic.

One of the goals for this app is to "future proof" it as much as possible. While that will never be 100% of course, I would like to set it up for when browsers natively support similar functionality with CSS (such as :has()).

Pagination for example...

Two ways to trigger(?)

1. On overflow/scroll or
2. counting the number of visible rows
3. or both?

For counting, if the number of visible rows are more than than the number of returned rows, the number of visible rows are divided into the number of returned rows (I'm sure the math is more complicated than that) and then assigns a class or attribute to the <body>, denoting the number of page links to show.

That's it. CSS will take care of the rest on my side.

(At least for what is to be shown/hidden. There is probably more going on in the background to track things in memory, etc... This is just the logic to hide/show things.)

The class or attribute could look something like .page01 or attribute data-page="1". Which signals my prestaged CSS to unhide one of five prestaged pagination links. If there are more than five pagination links, then we show an alternative pagination link layout (future).

So, if for example, there are 10 visible rows and 25 results returned, ACSS would assign the class .page02 or attribute data-page="2" to the <body>. The CSS has already been designed and coded (prestaged) to unhide two out of five possible five pagination links.

When it comes to pagination, this effectively separates the data logic (counting ACSS) and the UI logic (displaying the page links CSS).

Wonder if we can reliably prestage the pagination link URLs also - so there is less to do on the ACSS side?

Does that make sense?

I would like that prestaged two sided logic for everything having to do with hiding/showing DOM elements (if possible).

Hopefully that general overall strategy will make things easier and take some of the burden off both of us.

WDYT?

Forgive me if I'm preaching to the choir and you already have it all planned out. I just wanted to get that out there so at least I can better understand how we are going about this and are on the same page. These are just ideas and I am DEFINITELY open to better ones.

[dragontheory]

On the topic of finding another rich data set, I found this deepfake website from which we may be able to copy image addresses.

https://generated.photos/faces/right-facing/elderly/joy

And we could use the other JSON dataset of names and addresses? One of the things that was missing from that dataset were image URLs (URIs?).

Thoughts on datasets...

My first real business guinee pig for this app (outside of my work) is for my wife's business. She owns a ballet studio. Which means JSON data for families, relationships, children, siblings, one to many phone numbers, one to many email addresses, one to many addresses, credit card information, lessons taken, lessons schedules, lessons missed, point of sale/transaction data, etc. Some kids have one set a parents, some have no parents but have one to many guardians and still others have multiple sets of divorced parents with multiple sets of kids, with optional descriptions and or comments for everything.

This of course hints at what I am ultimately going to using this for.

Whatever dataset we use, it should emulate this sort of thing.

WDYT?

[bob2517]

I can do the page01 thing on the body, but as you say, it's not much use once it gets past 5 pages. A solution could be to just show next and previous links and skip the numbers altogether once it gets over 5 pages, and thereby encourage people to use the search instead. Is there going to be a search box?

I can't see how you can handle the large number of pages scenario with CSS alone. But do you have an idea for it?

Personally I rarely need to go beyond page 5, for anything, in practical terms. I would resort to search if it was that far back. So maybe something simple like a search box would be a solution. I can add that to the API functionality.

However you could prestage something fixed to handle large rows, like if you were on page 14, show 4 fixed links - "page 1 | page 13 | page 15 | last page", and then I can handle the logic to put in numbers based on the current page.

It would be good to make a decision on that by Saturday though, as I'd like to wrap up the whole pagination on that day if possible.

The infinite scrolling is good though. I can roll on that right now.

Re the one-to-many aspects of the later business use. That's quite a large system you're looking at. It would be good to see what the screen would ultimately look like to handle all that on one page alone. Only then could it be planned for and built. I've built more complex systems myself, but not all on one page...

[bob2517]

Re the setting of the page numbers - I am fetching a "totalRecords" variable from the API, which tells us how many pages we need.

Eg. If there are 10 rows on a page, and there are 543 record in the database, there will be 54 pages. There are several hundred records currently in the supe database.

Forgive me if I'm preaching to the choir and you already have it all planned out. I just wanted to get that out there so at least I can better understand how we are going about this and are on the same page. These are just ideas and I am DEFINITELY open to better ones.

It's a tricky one. I think the strategy has to be this.

1) You give me a full spec. I will build it. I can build anything with a full spec.

2) If you don't have a full spec in your mind, or only a half-solution, give me an end result and leave me alone to handle it in the best way that I know how. If later on you have a eureka moment, let me know and I can change it. I don't generally make future-proof mistakes, unless the spec I was given to build something is very incomplete, like only setting up one form UI, and then being told it needs to handle one-to-many databases and 10 admin forms in one page. That would be a ludicrous demand without a full detailed brief and screen plan at the outset.

3) Or if you don't have a full spec in your mind, don't ask me to do it, but give me something else that you do have a full spec for.

[bob2517]

I'm going to work on the infinite scroll for the moment, as that does have a final spec. I'll get it doing that by default, and await instructions on what toggle button we are going to have to switch to pagination.

I think we should probably take these up one at a time. Focus on a solution for one thing, then we can complete it, and then look at the next thing.

[bob2517]

That's the best way to get out of a confusing situation. Find one thing to focus on, complete it, then focus on the next thing, etc. Otherwise it will all get too overwhelming, as I imagine your brain is getting right now (insert meme).

[bob2517]

That's the way I built the core. I had a whole list of things to do, and I just put them into a sequence and completed them one by one. I only ever tackle one thing at a time. But I make sure that I complete it to its final product before I leave it. I never half-build anything with the intention of coming back to it later - that never gets anything done. I never start anything with a vague idea - I always map it out fully before I start. Even if it's just one piece of the overall puzzle.

This is why I want a full, full, full spec for the pagination before I start - I don't want to come back to it again once I've done the work on it, unless there is a bug with it.

[bob2517]

Tell you what, we'll do this, unless you get a better idea meanwhile:

1) We use the page 1 to 5 CSS logic that you specified. 2) I'll design a "greater than 5" page scenario set of elements to handle greater than 5, which will be static and work like your "page01" class in the body, but called something else. Actually the data-page="1" approach is better, so lets go with the attributes instead of the class. Anything not data-page=1 to 5 should hide the pages 1 to 5 links and show the links that I will create. 3) So I'll build that and then tell you where the elements are, so you can set up the CSS to hide your page 1 to 5 links and show the "greater than 5" elements. 4) Let me work out the rest and ask if I have questions. I've already got the behind-the-scenes logic for the actual pagination in my mind.

[bob2517]

I can build it without you doing your CSS, so there is no hold-up. Because I'll leave the 1 to 5 links up there on the branch. It'll just be up to you to hide/show them based on the value of the data-page attribute. Anything not 1 to 5 will show my links (which will have its own container) and hide your links, and vice versa.

At some point we should look at merging the branch into the master. My suggestion is to do that once I've got the pagination working. I'll convert tabs back to spaces before we do it.

[bob2517]

So the target is to have both infinite scroll and the pagination complete by the end of the weekend, and in a position to merge the branch back onto master by then too.

Then we can choose the next thing to do, or you can start working that out in the meantime. Consider the pagination a done deal from your side, with a final review from you and any amendments to do that you now see are needed, when I've finished coding it.

[bob2517]

That's lot of typing. Does that sound like a plan? Sorry to waffle on - I like to get these things straight in my head so I can see a smooth path.

[bob2517]

Once the pagination is completed it's final form, then I suggest we work out a strategy for replacing the API with something more professional like you mentioned. It will require a hosted database, like the supe one, which I can sort out when we know what the records are going to look like. We could even get the CRUD code working as a way to build the database up, rather than going through an import procedure. That may be a better sequence. It will be a good way to test it.

[bob2517]

For the CRUD database, I need to have a list of fields and any one-to-many relationships - I recommend at least one. Plus a way designed on paper to add and edit the related tables for a particular field. Plus an example of a dropdown, a radio button and a checkbox.

So we'll need the forms designed for that and you should make the call on what should be in there. These should be planned for before we start. Then you have a hope of getting your wife's system built.

It's actually not a lot of work to get the screen planned - it could just be done with pen on paper. That's usually how I would go about it.

[dragontheory]

Wow. Yes. Appreciate all of your thorough thought processes. I like to have things planned out as well.

Well, my wife's business is using a (terrible ) software package called Studio Director (thestudiodirector.com). Checking to see what their JSON looks like (if they use JSON).