search

cessda / cessda.cvs.two

Apache License 2.0
0 stars 2 forks source link

Some CVs have spurious version information #446

Closed cessda-bitbucket-importer closed 1 year ago

cessda-bitbucket-importer commented 2 years ago

Original report on BitBucket by John Shepherdson (GitHub: john-shepherdson).

TimeZone has a banner saying that a newer version is available, but the link does not work and it is not listed in the Versions tab.

‌

cessda-bitbucket-importer commented 2 years ago

Original comment by Taina Jääskeläinen.

It seems the system now displays a new version in the general UI even the CV version is only in some draft status in the Editor.

New version links should only apply to published vocabularies.

‌

cessda-bitbucket-importer commented 1 year ago

Original comment by Taina Jääskeläinen.

Issue #451 was marked as a duplicate of this issue.

cessda-bitbucket-importer commented 1 year ago

Original comment by Taina Jääskeläinen.

@pakoselo Why did you reopen the issue?

I checked and it seems resolved in staging, For instance, the vocabularies KungPao and TimeZone ‘en’ are in REVIEW in staging (3.1 and 1.1) but the general UI now shows only their published versions (3.0 and 1.0), with no ‘newer version available’ notification.

And ‘da’ version of AnalysisUnit now displays 2.0.1 in the UI with no notification about newer version even though it is in REVIEW 3.0.1 in the Editor.

So I will close unless you noticed something needing more amendment.

cessda-bitbucket-importer commented 1 year ago

Original comment by Martin Šeleng (GitHub: pakoselo).

@‌TainaFSD it is not working for 100%, It will not shown the info if you open old CV version. I will finnish it today.

cessda-bitbucket-importer commented 1 year ago

Original comment by Taina Jääskeläinen.

Ahh, great that you noticed - I would have missed that!

cessda-bitbucket-importer commented 1 year ago

Original comment by Martin Šeleng (GitHub: pakoselo).

@‌TainaFSD it is finished, but I notice there are errors in the vocabulary object in the database. The versions are Ok, but vocabulary object should work as a wrapper of all versions. It should hold latest number versions of all published languages, their definitions, source language, notation and publication dates. Actually it is not working for 100% as version numbers are not updated against the version table from the database. It is working for the new vocabularies, but not for every old vocabulary. Please see this one, which is created by me and working: https://vocabularies-dev.cessda.eu/vocabulary/111?lang=en&v=1.0 . It points you to new version which is 1.1 and it will not mesh with still ready to translate version 1.2 (you can see it in editor: https://vocabularies-dev.cessda.eu/editor/vocabulary/111?lang=en . But let’s have a look at the old ones: https://vocabularies-dev.cessda.eu/vocabulary/AnalysisUnit?lang=en&v=2.2 it shows, that the latest version is 2.1, which is bad (and of course misleading, because we are viewing the 2.2 already), but the link itself will work and points you to 2.4, which is actually the latest version. To solve the issue, it will be needed to hack the database and update all vocabularies with latest information from every language versions.

cessda-bitbucket-importer commented 1 year ago

Original comment by Taina Jääskeläinen.

Oh dear, that sound time-consuming. I seem to remember that at one point we had problems with inconsistent versions (before IISAS), so may date quite far back. As you resolved this issue, will you make another one for the database hacking?

As Martin B. is working on the new versioning design, I guess you are discussing this also with him. Although, I’m afraid this is the first time it has occurred to me to think about the old vocabularies once the new versioning is implemented. How to handle them? Can their versions remain language-specific or will they need to be changed to a three-digit version that is the same for all languages (i.e. new design)?

cessda-bitbucket-importer commented 1 year ago

Original comment by Martin Šeleng (GitHub: pakoselo).

Hi @‌TainaFSD I had a discussion with @Stifo , as he is working on the 3 digit system and not Martin and he propose not to edit old version numbers. They were deployed at that time and it was ok then, why we should change it. He just creates for SL versions a hack, where he adds the zero at the end. Only we need to make the vocabulary table consistent with the version tables, regarding the versions (mainly there were errors). But we can discuss it tomorrow.

‌

cessda-bitbucket-importer commented 1 year ago

Original comment by Taina Jääskeläinen.

The ‘newer version’ issue has reappeared.

‌

‌

Note: this vocabulary is no longer 1.1.0 as I used it for User Guide amendment screenshots.

cessda-bitbucket-importer commented 1 year ago

Original comment by Taina Jääskeläinen.

This problem seems to have gone away, could no longer repeat is. Closing the issue.

cessda-bitbucket-importer commented 1 year ago

Original comment by Martin Šeleng (GitHub: pakoselo).

@‌TainaFSD I am opening the issue again as it is still not solved for old vocabularies.

cessda-bitbucket-importer commented 1 year ago

Original comment by John Shepherdson (GitHub: john-shepherdson).

@‌dolinarm has taken over from Taina as Service Owner

cessda-bitbucket-importer commented 1 year ago

Original comment by Maja Dolinar.

@pakoselo could you point me to which old vocabularies you are reffering to? I cannot seem to find one that would produce this mistake.

cessda-bitbucket-importer commented 1 year ago

Original comment by Martin Šeleng (GitHub: pakoselo).

@‌dolinarm e.g. here: https://vocabularies-dev.cessda.eu/vocabulary/ModeOfCollection?lang=en&v=3.0 it informs about new version 3.1, but this is not true as the newest one is 4.0. Probably this is not affected in every vocabulary and probably only on dev and staging, because we didn't deployed the latest CVS software into the production yet. if we correct the database on the dev and staging and it will be ok with the latest version information, we can later update also the production database to be sure everything is OK. As the main language for every vocabulary is English, there are no such errors, but if some of the vocabularies will have source language different to english, it will start making errors (probably).

cessda-bitbucket-importer commented 1 year ago

Original comment by Martin Šeleng (GitHub: pakoselo).

@matthew-morris-cessda Is it possible to deploy content of the production database to the dev or staging? I want to test the update script to have in vocabulary table information about latest versions of every language.

cessda-bitbucket-importer commented 1 year ago

Original comment by Martin Šeleng (GitHub: pakoselo).

@matthew-morris-cessda @john-shepherdson @Joshocan I writing to everybody (sorry for spam), because I am not sure who is responsible for what.

First question: Is it possible to deploy content of the production database to the dev or staging? I want to test the update script to have in vocabulary table information about latest versions of every language. Also sub-question: Can we have latest production database dump, so I can test it also locally, please?

Second is not question, but solution. I prepare bash script for updating the vocabulary table filling the latest version info for every language. Please see attached. It is dependent on procedures (for handling 2/3 digit version numbers), which were provided by @Stifo (Stefan Dlugolinsky). I don’t know if they are deployed also on dev/staging, but I think yes.

#!/bin/bash

#if [ $# -lt 1 ]
#  then
#    echo "USAGE: ./update.sh [ID(s)]"
#    echo "       [ID(s)] represents the list of vocabulary ID(s) and it's optional."
#    exit
#fi

echo "Creating SQL functions ..."
SQL="
delimiter //
USE cvs//
CREATE FUNCTION canonizeVersionNumber(
    versionNumber varchar(20)
)
RETURNS varchar(255) CHARSET utf8mb4
    DETERMINISTIC
begin
    return 
        CONCAT(
            LPAD(SUBSTRING_INDEX(SUBSTRING_INDEX(if (versionNumberParts(versionNumber) = 3, versionNumber, concat(versionNumber, '.0')), '.', 1), '.', -1), 10, '0'),
            LPAD(SUBSTRING_INDEX(SUBSTRING_INDEX(if (versionNumberParts(versionNumber) = 3, versionNumber, concat(versionNumber, '.0')), '.', 2), '.', -1), 10, '0'),
            LPAD(SUBSTRING_INDEX(SUBSTRING_INDEX(if (versionNumberParts(versionNumber) = 3, versionNumber, concat(versionNumber, '.0')), '.', 3), '.', -1), 10, '0')
        );
end//

CREATE FUNCTION versionNumberParts(
    versionNumber varchar(20)
) RETURNS int
    DETERMINISTIC
begin
    return length(versionNumber) - length(replace(versionNumber, '.', '')) + 1;
end//
"
echo "$SQL" | mysql

LANGS="bg bs cs da de el en es et fi fr hr hu it ja lt mk nl no pl pt ro ru sk sl sq sr sv"
if [ $# -lt 1 ]
then
    ARR=$(echo "USE cvs; SELECT id FROM vocabulary" | mysql --skip-column-names)
else
    ARR=$(echo $1 | sed "s/'/\\\'/g")
fi

for LANG in $LANGS
do
    echo "Processing language" $LANG
    for ID in $ARR
    do
        echo "Processing vocabulary" $ID
        SQL="USE cvs;
            UPDATE vocabulary
            SET version_number=
                (select number_mapping.number
                from
                    (select vocabulary_id, item_type, language, max(id) as latest_version_id from version where status='PUBLISHED' group by vocabulary_id, item_type, language) latest_versions,
                    (select vocabulary_id, max(canonizeVersionNumber(number)) as number_latest_canonical from version where status='PUBLISHED' group by vocabulary_id) latest_version_ids,
                    (select distinct(number), canonizeVersionNumber(number) as number_canonincal from version) number_mapping,
                    (select source_language, id from vocabulary) source_language_mapping
                where
                    latest_versions.vocabulary_id=latest_version_ids.vocabulary_id and
                    latest_version_ids.number_latest_canonical=number_mapping.number_canonincal and
                    source_language_mapping.source_language = latest_versions.language and
                    source_language_mapping.id = latest_versions.vocabulary_id and
                    source_language_mapping.id='"$ID"')
            where 
                vocabulary.id='"$ID"';

            UPDATE vocabulary
            SET version_"$LANG"=
                (select number_mapping.number
                from
                    (select vocabulary_id, max(canonizeVersionNumber(number)) as number_latest_canonical from version where status='PUBLISHED' and language='"$LANG"' group by vocabulary_id) latest_version_ids,
                    (select distinct(number), canonizeVersionNumber(number) as number_canonincal from version) number_mapping
                where
                    latest_version_ids.number_latest_canonical=number_mapping.number_canonincal and
                    latest_version_ids.vocabulary_id='"$ID"')
            where 
                vocabulary.id='"$ID"';
                "
        #printf "%s $SQL"
        echo "$SQL" | mysql
    done
done

echo "Droping functions ..."
SQL="
USE cvs;
DROP FUNCTION IF EXISTS canonizeVersionNumber;
DROP FUNCTION IF EXISTS versionNumberParts;
"
echo "$SQL" | mysql

‌

‌

cessda-bitbucket-importer commented 1 year ago

Original comment by Stefan Dlugolinsky (GitHub: Stifo).

@pakoselo nope, the procedures were not deployed there. it was mine local helper stuff.

cessda-bitbucket-importer commented 1 year ago

Original comment by Martin Šeleng (GitHub: pakoselo).

@Stifo OK, we would need to change it somehow, to be executable also without the functions.

cessda-bitbucket-importer commented 1 year ago

Original comment by Joshua Tetteh Ocansey (GitHub: Joshocan).

@pakoselo I sent you a copy of the production database. Feel free to test the script on it on your local instance.

Let us know if there is anything else we can help you with.

cessda-bitbucket-importer commented 1 year ago

Original comment by Martin Šeleng (GitHub: pakoselo).

@Joshocan thanks for that, another thing is to replicate the production DB to staging or dev, to be able to test all main features and migration scripts on real data (as the dev and staging databases are little bit weird). Probably it will be good idea to remove unnecessary users and add iisas_guest there please.

cessda-bitbucket-importer commented 1 year ago

Original comment by Martin Šeleng (GitHub: pakoselo).

@‌dolinarm I have tested the script on the latest DB provided by @Joshocan locally on my environment and it looks good, so later we can deploy the script on the dev or/and staging environments too.

cessda-bitbucket-importer commented 1 year ago

Original comment by Maja Dolinar.

@pakoselo great news! Let me know when you deploy so that I can test it.

cessda-bitbucket-importer commented 1 year ago

Original comment by Martin Šeleng (GitHub: pakoselo).

@Joshocan Can you please deploy the production DB into the dev environment?

cessda-bitbucket-importer commented 1 year ago

Original comment by Joshua Tetteh Ocansey (GitHub: Joshocan).

@pakoselo Sorry, I missed this message. Why do you want production DB in dev?

You will not be able to login into dev after exporting.

cessda-bitbucket-importer commented 1 year ago

Original comment by Martin Šeleng (GitHub: pakoselo).

@Joshocan Please see above. I kindly ask you also to add iisas_guest user and remove users which are not needed. We need to refresh the dev/staging environments because we and Taina were testing new features and the DB is really broken.

cessda-bitbucket-importer commented 1 year ago

Original comment by Joshua Tetteh Ocansey (GitHub: Joshocan).

@pakoselo A copy of production DB is now in dev.

You Have created an account for IISAS_guest; an activation email was sent to you. You should be able to log in into https://vocabularies-dev.cessda.eu/ as an admin.

Let me know if there is anything else you need.

‌

cessda-bitbucket-importer commented 1 year ago

Original comment by Martin Šeleng (GitHub: pakoselo).

@Joshocan Are you sure, there is a mail server installed on the dev? I think, there is no mail server. I didn’t receive anything, if you use my email, it didn’t arrive. Can you please set there our old password “uisav05”?

cessda-bitbucket-importer commented 1 year ago

Original comment by Joshua Tetteh Ocansey (GitHub: Joshocan).

@pakoselo I tested the mail service, and it works I just received a mail for the CVS service.

Please check your spam in case. I used this email address [email address removed](mailto:email address removed)

cessda-bitbucket-importer commented 1 year ago

Original comment by Joshua Tetteh Ocansey (GitHub: Joshocan).

@pakoselo Do you still want to reset the password? In that case, I have to change the email to mine.

cessda-bitbucket-importer commented 1 year ago

Original comment by Martin Šeleng (GitHub: pakoselo).

@Joshocan Sorry, I was away. It finaly arrived, but how did you wipe the DB, because I still see there lot of junk vocabularies.

‌

cessda-bitbucket-importer commented 1 year ago

Original comment by Joshua Tetteh Ocansey (GitHub: Joshocan).

@pakoselo You have to reindex ES from the Maintenance page to load the load. I just reindex ES.

‌

cessda-bitbucket-importer commented 1 year ago

Original comment by Martin Šeleng (GitHub: pakoselo).

Yes, I do the same.

cessda-bitbucket-importer commented 1 year ago

Original comment by Joshua Tetteh Ocansey (GitHub: Joshocan).

@pakoselo CVS dev is updated but with some errors such as:

Processing vocabulary 94
Processing language sv
Processing vocabulary 82
ERROR 1048 (23000) at line 2: Column 'version_number' cannot be null
Processing vocabulary 83
Processing vocabulary 84
ERROR 1048 (23000) at line 2: Column 'version_number' cannot be null
Processing vocabulary 86
ERROR 1048 (23000) at line 2: Column 'version_number' cannot be null
Processing vocabulary 87
Processing vocabulary 88
ERROR 1048 (23000) at line 2: Column 'version_number' cannot be null
Processing vocabulary 90
ERROR 1048 (23000) at line 2: Column 'version_number' cannot be null
Processing vocabulary 94
Droping functions ..

I hope it is expected behavior.

@‌dolinarm You may be able to test now.

cessda-bitbucket-importer commented 1 year ago

Original comment by Martin Šeleng (GitHub: pakoselo).

Yeah, but you forget to execute the new one in the new issue, created by Martin Bobak. As there is one additional line for removing the translator_admin or what role.

cessda-bitbucket-importer commented 1 year ago

Original comment by Joshua Tetteh Ocansey (GitHub: Joshocan).

@pakoselo Yes, It is updated now.

@‌dolinarm Let us know the outcome of the testing.

cessda-bitbucket-importer commented 1 year ago

Original comment by Maja Dolinar.

Will do!

cessda-bitbucket-importer commented 1 year ago

Original comment by Maja Dolinar.

@Joshocan there seems to be an issue in accessing the service on dev: https://vocabularies-dev.cessda.eu/

I get 503 Service Unavailable message

cessda-bitbucket-importer commented 1 year ago

Original comment by Joshua Tetteh Ocansey (GitHub: Joshocan).

@‌dolinarm It works fine as of this morning. Let me know if see differently from the image below.

‌

cessda-bitbucket-importer commented 1 year ago

Original comment by Maja Dolinar.

As of now I can no longer access the User management as well, I can only access the published CVs and user guides, going to Editor or User management gives me an Error page!

i have to say that my access keeps changing, yesterday when I waited for a couple of hours it came back. Today in the morning I could access User Management and Editor, but could not add any new CVs. Now I cannot access anything anymore in terms of editing. Please have a look at this, it is really impossible for me to test things. @Joshocan

cessda-bitbucket-importer commented 1 year ago

Original comment by Joshua Tetteh Ocansey (GitHub: Joshocan).

@pakoselo Do you get the above error in your local instance when you try to “add a CV”?

I see this log message 

2023-01-24 09:46:45.922 CET
Unauthorized: Full authentication is required to access this resource
Open in Logs Explorer
{
insertId: "9cyghgekf5zs6fkb"
jsonPayload: {7}
labels: {6}
logName: "projects/cessda-dev/logs/stdout"
receiveTimestamp: "2023-01-24T08:46:49.667852140Z"
resource: {2}
severity: "WARNING"
timestamp: "2023-01-24T08:46:45.922997519Z"
}
2023-01-24 09:46:49.171 CET
Unauthorized: Full authentication is required to access this resource
2023-01-24 09:46:58.568 CET
Unauthorized: Full authentication is required to access this resource
2023-01-24 09:47:24.774 CET
Unauthorized: Full authentication is required to access this resource

By the way, I have a Role_Admin right

cessda-bitbucket-importer commented 1 year ago

Original comment by Joshua Tetteh Ocansey (GitHub: Joshocan).

Apparently, It is the same behavior in production. How did we miss this?

I have created a separate issue on it. @‌Martin Bobak @pakoselo @Stifo Please anyone take a look at this #466 as matter urgency.

cessda-bitbucket-importer commented 1 year ago

Original comment by Joshua Tetteh Ocansey (GitHub: Joshocan).

@‌dolinarm Can you proceed with testing other functionality without #466?

cessda-bitbucket-importer commented 1 year ago

Original comment by Maja Dolinar.

@Joshocan unfortunately not, since I currently cannot access the Editor. I just receive Error pages with unauthorized access.

cessda-bitbucket-importer commented 1 year ago

Original comment by Martin Bobak.

@‌dolinarm even if you are ADMIN_SL or ADMIN_TL? Those two roles are working correctly in my local instance. You are right about ADMIN_CONTENT - it looks like there is a conflict between role ADMIN and ADMIN_CONTENT.

cessda-bitbucket-importer commented 1 year ago

Original comment by Maja Dolinar.

@‌Martin Bobak I cannot test this currently, because I am not allowed anymore to access the Editor on Dev.

cessda-bitbucket-importer commented 1 year ago

Original comment by Joshua Tetteh Ocansey (GitHub: Joshocan).

@‌Martin Bobak I am having the same issue even though I have a ROLE_ADMIN role.

cessda-bitbucket-importer commented 1 year ago

Original comment by Martin Bobak.

Even with your account register on gmail? That user is a ADMIN_TL for Slovenian language within agencies CESSDA and DDI Alliance.

cessda-bitbucket-importer commented 1 year ago

Original comment by Maja Dolinar.

@‌Martin Bobak yes with ROLE_USER and ADMIN_TL on my other account I can access. It doesn’t seem to be working for ADMIN_CONTENT.

cessda-bitbucket-importer commented 1 year ago

Original comment by Martin Bobak.

@‌dolinarm yes the problem occurs only with ADMIN_CONTENT, but the other roles are working correctly, do not they?

cessda-bitbucket-importer commented 1 year ago

Original comment by Martin Bobak.

@Joshocan since @‌dolinarm has two accounts can you assign to one of them ROLE_ADMIN (e.g. registered on gmail while it does not work) and the other will be for role testing? Thus she is able to change its user role as she needs.

@pakoselo we are discussing the issue #466 here.

‌