add short def to wordlist export

[balmas]

(in user target language when we have support for that)

requested by Matteo Romanello

[irina060981]

@balmas , would it be enough this way - https://github.com/alpheios-project/alpheios-core/pull/232

word    definition
beatum (beatus) happy, fortunate, bringing happiness; rich, wealthy, copious, sumptuous;
beatum (beatus) blessed, blissful; "Saint" (in early Church, less formal);
beatum (beo)    bless, make happy, gladden, delight; enrich (with);
beatum (beatus) happy/fortunate men/persons (pl.); "the_rich"; The Blessed, Saints;
beatum (beatum) happiness, blessedness; good fortune; good circumstances;
placito (placeo)    please, satisfy, give pleasure to (with dat.);
placito (placitus)  pleasing; agreed upon;
placito (placet)    it is pleasing/satisfying, gives pleasure; is believed/settled/agreed/decided;
placito (placitum)  plea;
placito (placito)   plead;

Or should we add all definitions from all lexeme to one line?

[balmas]

Ideally, I think we would have each lexeme on its own line, as you have here, and if a lexeme's lemma has principal parts, we should include them, similar to the way we do in the popup. so, in the above example, for beatum, the output would be:

`
word    definition
beatum (beatus, beata, beatior, beatissimus)    happy, fortunate, bringing happiness; rich, wealthy, copious, sumptuous;
beatum (beatus, beata, beatum)  blessed, blissful; "Saint" (in early Church, less formal);
beatum (beo, beare, beavi, beatus)  bless, make happy, gladden, delight; enrich (with);
beatum (beatus, beati)  happy/fortunate men/persons (pl.); "the_rich"; The Blessed, Saints;
beatum (beatum, beati)  happiness, blessedness; good fortune; good circumstances;

This information, as well as the short definition, can be obtained from the full homonym, but we don't store that on the remote server.

Although the user may have a local store of the entire wordlist, they may not, so we have to decide what to do about that. We can either

(1) always store the entire homonym on the remote server (2) store only the additional parts we need for quizlet download (i.e. principal parts and short definitions) on the remote server (3) if a quizlet download has been requested, for any word on the list which doesn't have a full homonym available in the local cache, then we first populate via lookup and then download the list. (4) same as (3) but limit the quizlet download to user-selected set of max X number of words, so that we aren't processing thousands of words for download.

I am a little worried about option (1), storing the full homonym on the remote server, because I think it will cause us to exceed the size requirements of the dynamodb api we are using, and because it will in many cases be unnecessary if users don't use this feature. I have similar concerns about (2), particularly because many of the short definitions are not that short.

I'm tempted to go with (4) to start. I think typically quizlet decks are fairly short, and it might be preferable anyway to restrict this to a subset.

@mromanello do you have any thoughts on this? Would it be reasonable and make sense from a student/teacher perspective to limit the wordlist export for quizlet to a subset of the terms that the user selects before download?

[irina060981]

The first step is done with minimal requirements from #201

word    definition
beatum (beatus, beata, beatior, beatissimus)    happy, fortunate, bringing happiness; rich, wealthy, copious, sumptuous;
beatum (beatus, beata, beatum)  blessed, blissful; "Saint" (in early Church, less formal);
beatum (beo, beare, beavi, beatus)  bless, make happy, gladden, delight; enrich (with);
beatum (beatus, beati)  happy/fortunate men/persons (pl.); "the_rich"; The Blessed, Saints;
beatum (beatum, beati)  happiness, blessedness; good fortune; good circumstances;
mare (mare, maris)  sea; sea water;
mare (mas, maris)   male (human/animal/plant); man;
mare (mas, (gen.), maris)   male; masculine, of the male sex; manly, virile, brave, noble; G:masculine;
mare (mare) male; masculine, of the male sex; manly, virile, brave, noble; G:masculine;

[mromanello]

Hello @balmas and @irina060981

First off, it's great to see how rapidly you've made progress on this issue! Many thanks for considering my request. Concerning your question:

@mromanello do you have any thoughts on this? Would it be reasonable and make sense from a student/teacher perspective to limit the wordlist export for quizlet to a subset of the terms that the user selects before download?

In a teaching scenario, I can think of the following selection criteria:

• limit export to words in source language X (e.g. Arabic), thus excluding for example those in Latin and Greek;
• limit export to the n most recently looked up words (good for regular terminology review cycles);
• limit export to lemmas found in following works or in works by following authors (this could be useful for courses that focus extensively on a single author)
• limit export to the most often looked up words (but here I don't know whether you store such information in Alpheios).

Hope this helps somehow!

[balmas]

Thanks @mromanello ! These use cases imply a few new requirements for the wordlist, but all of them they are on our roadmap so it should be possible to cover them.

I think we may also need to add a max number of items for download regardless of which of the above options is picked.

[balmas]

I think of the requests, limit to authors/works is the hardest and might have to wait a bit longer for implementation.

[balmas]

limit export to words in source language X (e.g. Arabic), thus excluding for example those in Latin and Greek;

this is already there

[balmas]

limit export to the n most recently looked up words (good for regular terminology review cycles);

this requires #241

[balmas]

limit export to the most often looked up words

this requires #240

[balmas]

limit export to lemmas found in following works or in works by following authors (this could be useful for courses that focus extensively on a single author)

this requires #242

[balmas]

@irina060981 i think for the upcoming release, I would like to add the following features:

1. a configurable max items for flashcard export (I do not wish to expose this setting to the end user in options, however). Let's set it to 25 to start.

2. an option to export only the X most recently added words on the list , where X is user-specifiable and limited to the max value when exporting for flashcards.

When exporting items for flashcards, for any word on the list which doesn't have a full homonym available in the local cache, then we first populate the homonym via lookup for downloading the list.

The other requests above will be targeted for the next release (with the exception of author/work which may need to wait longer). Let me know if this doesn't make sense to you.

[irina060981]

@balmas , I have started to work on this issue I have faced with the following error trying to add the first word

PUT https://userapis.alpheios.net/v1/words/lat-socium

returns 500 error (for the existed Google user - irisha060981@gmail.com)

[irina060981]

The same error for another account - irina060981@yandex.ru

PUT https://userapis.alpheios.net/v1/words/lat-ostium

returns 500 error

[irina060981]

I have found my account on AWS, that you gave me before, but I didn't find here - where I could see logs - to find what is the exactly error for the saving the word to remote. @balmas , could you help me with it?

[balmas]

Ah, the problem is that we need to update the schema used by the service to validate the new fields.

https://github.com/alpheios-project/user-word-api/blob/master/schema.json

If you can send me a PR for the schema change, I'll put it up in the dev environment so that you can use that.

[balmas]

@irina060981 the new schema has been deployed at userapis-dev.alpheios.net

[irina060981]

@balmas , there is still a problem with saving frequency - it is not saved (it stays undefined in remote) and returns 500 Error

This is object sent to remote:

ID: "lat-crederetur"
context: [{…}]
createdDT: "2020/04/21 @ 17:28:05"
frequency: 2
homonym: {targetWord: "crederetur", lemmasList: "credo"}
important: false
languageCode: "lat"
listID: "google-oauth2|114076270674179873629-lat"
targetWord: "crederetur"
updatedDT: "2020/04/21 @ 17:36:26"
userID: "google-oauth2|114076270674179873629"

[balmas]

I am not sure what is in the context element, but I have tested the following object and it is working for me in the dev environment:

{
  "ID": "lat-crederetur",
  "listID": "google-oauth2|114076270674179873629-lat",
  "userID": "google-oauth2|114076270674179873629",
  "languageCode": "lat",
  "targetWord": "crederetur",
  "important": false,
  "createdDT": "2020/04/21 @ 17:28:05",
  "frequency": 2,
  "homonym": {
    "targetWord": "crederetur",
    "lemmasList": "credo"
  },
  "context": [],
  "updatedDT": "2020/04/21 @ 17:36:26"
}

Note that I did make a change to the schema this morning to make the new fields optional. I think we cannot make them required until all of the old code is out of use.

[irina060981]

Realized the following:

1. Added new fields for worditem

   • updatedDT
   • frequency

2. Added additional filter items

   • most recent (by updatedDT)
   • most often (by frequency)

3. each additional filter has filterAmount field, it defines amount of words in filter

   it gets default data from featureOptioins.wordlistFilterAmountDefault

4. added another featureOptions.wordlistMaxFlashcardExport option

5. On flashcard export if worditem doesn't have shortDefs - it inits additional lexicalQuery and download the wordlist only after all lexical requests are complete while lexical data is loading - progressbar is shown

6. also added frequency and updatedDT to wordlist ui and added sorting for them

[irina060981]

I believe that for other requirements we have other issue. @balmas , what do you think - could we close it?

[balmas]

I believe that for other requirements we have other issue. @balmas , what do you think - could we close it?

Yes!

[balmas]

Alpheios Components 3.3.0-qa.20200501441

@monzug to test this you're going to need to use the dev url for the user-word api in your env-webext.js file:

'wordlist' : 'https://userapis-dev.alpheios.net/v1/words',

[balmas]

@monzug also, when testing the wordlist changes, can you provide an opinion on whether or not you think more explanatory information is needed for the user about what the new columns and filter options are? @abrasax and I discussed maybe doing this via a help popup from the panel, but I'm not positive if it's necessary or not.

[monzug]

I updated the env-webext.js (that might explain why I was getting an empty wordlist while testing https://github.com/alpheios-project/alpheios-core/issues/291). I haven't tested the functionality yet, these are my feeling on the UI: 1) login as an existing user, all saved words from the old times, have the today date. I would rather see the today date not being applied on existing wordlist. it's misleading. 2) the arrows should have a tooltip 3) arrows are too close to each others - see screenshot below 4) when I change from most recent (or alphabetical order) to most often, the order of the words changes, even if they are all 1. Is the frequency of the word based on the user frequency or from an external database? in my scenario, philosophiae is always at the top of the most often words in my list. Note that when I change direction of the most often arrows, nothing changes: philosophiae is always at the top. How should these arrows work? see comment below from today (5/18) 5) question: why the short def is not longer offered after I logged out? is it supposed to be once per session? 6) I am not sure how the limit to most often and most recent should work: if I have a number saved from previous limit, e.g. 4, but in the meantime I added some words to the wordlist and it's more than 4 words, the limit to 4 is not applied. it does display all words in wordlist. 7) when I switch from limit by this session (or important) to most often or most recent, it does not work. I have to change the number of words to show in order to have the most recent or most often list. 8) also, when I limit to most often to 5 words and I have one word with frequency of 3 and one word with frequency of 2, plus 3 words with frequency 1. when I change directions of the most often arrows, only the word with frequency 3 and the one with frequency 2 change position. see comment below from today (5/18)

[monzug]

finally I got that when arrows are all blacks (not red arrows), it goes back to original order. got it! so number 8 is not a question/issue.

[balmas]

login as an existing user, all saved words from the old times, have the today date. I would rather see the today date not being applied on existing wordlist. it's misleading.

This is a tough one. @irina060981 to fix this I guess we would have to not add an update date to a wordlist item if it didn't already have one. Do you think it's doable?

[balmas]

the arrows should have a tooltip

The problem with tooltips is that they don't work on mobile. So I think we probably do need to have a popup help for the wordlist then. I will create an issue for that.

[balmas]

question: why the short def is not longer offered after I logged out? is it supposed to be once per session?

I would guess this is because of the way we broke things with the events as discussed in #291 so should be retested when that is fixed

[balmas]

@irina you investigate the other items mentioned here? Thanks!

[monzug]

I will resume my testing on these changes when #291 is fixed.

On Mon, May 18, 2020 at 10:28 AM Bridget Almas notifications@github.com wrote:

@Irina https://github.com/Irina you investigate the other items mentioned here? Thanks!

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/alpheios-project/alpheios-core/issues/200#issuecomment-630219469, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJ32UONB64JP6IBUTWK2JE3RSFAXPANCNFSM4LNV7IMA .

[monzug]

as per #3 above in the list, when I am in http://sedra.bethmardutho.org/library there is no way that I can select the UP arrow for most often. the down arrow yes, the up and down arrow for most recent but not the UP arrow for most often.

[irina060981]

as per #3 above in the list, when I am in http://sedra.bethmardutho.org/library there is no way that I can select the UP arrow for most often. the down arrow yes, the up and down arrow for most recent but not the UP arrow for most often.

This would be fixed with the following #364

[irina060981]

1. login as an existing user, all saved words from the old times, have the today date. I would rather see the today date not being applied on existing wordlist. it's misleading.

This is a tough one. @irina060981 to fix this I guess we would have to not add an update date to a wordlist item if it didn't already have one. Do you think it's doable?

I think we should change logic a little because we update item several times I removed overabundant changes of updatedDT and createdDT in the PR #374

[irina060981]

2. the arrows should have a tooltip

moved to a separate issue - #345

[irina060981]

3. arrows are too close to each others - see screenshot below

fixed in #364

[irina060981]

4. when I change from most recent (or alphabetical order) to most often, the order of the words changes, even if they are all 1. Is the frequency of the word based on the user frequency or from an external database? in my scenario, philosophiae is always at the top of the most often words in my list. Note that when I change direction of the most often arrows, nothing changes: philosophiae is always at the top. How should these arrows work? see comment below from today (5/18)

@monzug , as I could see in the code we support 2-properties sorting

first would be used that sorting that you clicked - for example frequency and at the second step would be used another sorting - next in the following order

sortingState: {
   'targetWord': null,
   'frequency': null,
    'updatedDT': null 
}

so it would be sorted by frequency and updatedDT if click on frequency by updatedDT and targetWord if click on updatedDT

[irina060981]

5. question: why the short def is not longer offered after I logged out? is it supposed to be once per session?

it is already fixed with defaultWordlistMaxFlashcardExport in #363

[irina060981]

6. I am not sure how the limit to most often and most recent should work: if I have a number saved from previous limit, e.g. 4, but in the meantime I added some words to the wordlist and it's more than 4 words, the limit to 4 is not applied. it does display all words in wordlist.

oh, i have missed such a case - would fix - would clear limitTo number after clearing the filter

[irina060981]

7. when I switch from limit by this session (or important) to most often or most recent, it does not work. I have to change the number of words to show in order to have the most recent or most often list.

@monzug , I didn't completely understand, the workflow here is the following

• you select limit by important - it filters the wordlist by important
• you select for example most recent - it applies the filter mostRecent with default value of amount
• you could change filterAmount and limited results would be updated

It is working for me this way

what steps are not working in your case?

[irina060981]

@balmas , I have reviewed all points - and besides couple of them (that have questions from me) all are in PRs #363, #364 and #374

[monzug]

let me try #7 again. it's the limit that was not working

[irina060981]

3,5,7 - fixed and merged with #363 and #364

[monzug]

irina, has issue #6 be moved to issue #374 or it's still part of this issue?

[irina060981]

6 I am not sure how the limit to most often and most recent should work: if I have a number saved from previous limit, e.g. 4, but in the meantime I added some words to the wordlist and it's more than 4 words, the limit to 4 is not applied. it does display all words in wordlist.

oh, i have missed such a case - would fix - would clear limitTo number after clearing the filter

Monica, do you mean this 6 ?

it would be fixed with https://github.com/alpheios-project/alpheios-core/pull/374 soon

[monzug]

yes. Thanks

as per #7, I was able to repro it again in Safari. have a wordlist with important words too. limit by this session, then switch to limit by most recent, then by limit by most often, the list does not change. only when I change the number of words to show from 25 to 24, the list get updated. let me know if you can repro.

[irina060981]

ok - with 7. I was able to reproduce - will fix tomorrow - thank you for detailed description!

[monzug]

No problem!

On Fri, May 22, 2020 at 2:54 PM Sklyarova Irina notifications@github.com wrote:

ok - with 7. I was able to reproduce - will fix tomorrow - thank you for detailed description!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/alpheios-project/alpheios-core/issues/200#issuecomment-632676185, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJ32UOLEFRFEECK6F7LXE73RSZYZ5ANCNFSM4LNV7IMA .

[irina060981]

1 and 6 was fixed with #374

[irina060981]

I have only 7 point of the issue to be not ready

[monzug]

Irina, on latest build I have noticed again that the number of words to show does not reset when when switching from one limit to an other. I know you are still working on this issue, so keep this screenshot as an other example.