Greenlight v2 => v3 migration script fails

[zulf10]

Hello,

I have a system setup with V2 and V3 and I am trying to migrate my roles, users etc. BBB is on 2.7.

Greenlight (v2 and v3) docker already in last version.

.env V2

SECRET_KEY_BASE=123
BIGBLUEBUTTON_ENDPOINT=https://XXX/bigbluebutton/ 
V3_ENDPOINT=https://XXX/bigbluebutton/ 
V3_SECRET_KEY_BASE=123

.env V3

BIGBLUEBUTTON_ENDPOINT=https://XXX/bigbluebutton/ 
SECRET_KEY_BASE=123

Command to migrate rôle: docker exec -it greenlight-v2 bundle exec rake migrations:roles

/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:68: warning: already initialized constant Net::ProtocRetryError
/usr/local/lib/ruby/2.7.0/net/protocol.rb:66: warning: previous definition of ProtocRetryError was here                                                                                    /usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:208: warning: already initialized constant Net::BufferedIO::BUFSIZE
/usr/local/lib/ruby/2.7.0/net/protocol.rb:206: warning: previous definition of BUFSIZE was here                                                                                            /usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:504: warning: already initialized constant Net::NetPrivate::Socket
/usr/local/lib/ruby/2.7.0/net/protocol.rb:503: warning: previous definition of Socket was here
warning: parser/current is loading parser/ruby27, which recognizes2.7.6-compliant syntax, but you are running 2.7.7.
Please see https://github.com/whitequark/parser#compatibility-with-ruby-mri.
Unable to migrate Role:                                                                                                                                                                    
ID: 6
Name: Moderateur
Errors: Provider does not exist
Unable to migrate Role:
ID: 7
Name: Basic
Errors: Provider does not exist
Unable to migrate Role:
ID: 8
Name: Membre niv. 2
Errors: Provider does not exist

Roles migration complete.
In case of an error please retry the process to resolve.

Nginx logs show 400 code

172.24.0.3 - - [08/Mar/2024:23:51:21 +0100] "POST /api/v1/migrations/roles.json HTTP/1.1" 400 47 "-" "Ruby"
172.24.0.3 - - [08/Mar/2024:23:51:21 +0100] "POST /api/v1/migrations/roles.json HTTP/1.1" 400 47 "-" "Ruby"
172.24.0.3 - - [08/Mar/2024:23:51:21 +0100] "POST /api/v1/migrations/roles.json HTTP/1.1" 400 47 "-" "Ruby"

Docker Greenlight-v3 logs same status 400

greenlight-v3    | I, [2024-03-08T22:51:21.636704 #1]  INFO -- : [a2cc9721-705b-41f4-b7aa-06638d3497f4] method=POST path=/api/v1/migrations/roles.json format=json controller=Api::V1::Migrations::ExternalController action=create_role status=400 allocations=1549 duration=8.18 view=0.60 db=4.16 time=2024-03-08 22:51:21 UTC host=
greenlight-v3    | I, [2024-03-08T22:51:21.649238 #1]  INFO -- : [9d593cf2-6ca7-4419-b161-1066a4d6a8ef] method=POST path=/api/v1/migrations/roles.json format=json controller=Api::V1::Migrations::ExternalController action=create_role status=400 allocations=510 duration=1.78 view=0.34 db=0.43 time=2024-03-08 22:51:21 UTC host=
greenlight-v3    | I, [2024-03-08T22:51:21.660275 #1]  INFO -- : [7bc57e43-c72e-4fa3-9e51-6a69ec40eb98] method=POST path=/api/v1/migrations/roles.json format=json controller=Api::V1::Migrations::ExternalController action=create_role status=400 allocations=507 duration=2.58 view=0.50 db=0.62 time=2024-03-08 22:51:21 UTC host=

Is there any special configuration for these errors?

It's as if API V1 is no longer available. Do I need to backtrack to BBB version 2.6 or even 2.5?

Thanks

[farhatahmad]

What's the command that you ran?

[zulf10]

I used: docker exec -it greenlight-v2 bundle exec rake migrations:roles

[farhatahmad]

Can you try updating both v2 and v3 please

[zulf10]

Yes, the containers are in the latest version and so is bbb.

[farhatahmad]

What's the output of

docker exec -it greenlight-v2 rails c
Role.all.pluck(:provider)

[zulf10]

It seems there is no rails command:

# docker exec -it greenlight-v2 rails c
OCI runtime exec failed: exec failed: unable to start container process: exec: "rails": executable file not found in $PATH: unknown

[farhatahmad]

Try either bin/rails or bundle exec rails

[zulf10]

Thanks for reply.

with command : # docker exec -it greenlight-v2 bin/rails c And : irb(main):001:0> Role.all.pluck(:provider)

=> ["greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight"]

[farhatahmad]

Thats very strange - everything looks good from here. re-running it will give you the same errors?

[zulf10]

Yes migration role command docker exec -it greenlight-v2 bundle exec rake migrations:roles give same errors:

/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:68: warning: already initialized constant Net::ProtocRetryError
/usr/local/lib/ruby/2.7.0/net/protocol.rb:66: warning: previous definition of ProtocRetryError was here
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:208: warning: already initialized constant Net::BufferedIO::BUFSIZE
/usr/local/lib/ruby/2.7.0/net/protocol.rb:206: warning: previous definition of BUFSIZE was here
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:504: warning: already initialized constant Net::NetPrivate::Socket
/usr/local/lib/ruby/2.7.0/net/protocol.rb:503: warning: previous definition of Socket was here
warning: parser/current is loading parser/ruby27, which recognizes2.7.6-compliant syntax, but you are running 2.7.7.
Please see https://github.com/whitequark/parser#compatibility-with-ruby-mri.
Unable to migrate Role:
ID: 6
Name: Moderateur
Errors: Provider does not exist
Unable to migrate Role:
ID: 7
Name: Basic
Errors: Provider does not exist
Unable to migrate Role:
ID: 8
Name: Membre niv. 2
Errors: Provider does not exist

Roles migration complete.
In case of an error please retry the process to resolve.

Is normal command Role.all.pluck(:provider) return many greenlight ?

=> ["greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight", "greenlight"]

isn't there a problem with the database that would cause this kind of error?

[farhatahmad]

Nope that's fine. Can you confirm your v2 and v3 versions please?

[zulf10]

• For v2

# docker image inspect bigbluebutton/greenlight:v2 | grep VERSION_CODE

"VERSION_CODE=release-2.14.9"

• For v3

# docker image inspect bigbluebutton/greenlight:v3 | grep VERSION_TAG

"VERSION_TAG=release-3.3.1"

• BBB package version with command

# dpkg -l | grep bbb

ii  bbb-apps-akka                         1:2.7-61                                     all          BigBlueButton Apps (Akka)
ii  bbb-config                            1:2.7-34                                     amd64        BigBlueButton configuration utilities
ii  bbb-etherpad                          1:2.7-14                                     amd64        The EtherPad Lite components for BigBlueButton
ii  bbb-export-annotations                1:2.7-19                                     amd64        BigBlueButton Export Annotations
ii  bbb-freeswitch-core                   2:2.7-15                                     amd64        BigBlueButton build of FreeSWITCH
ii  bbb-freeswitch-sounds                 1:2.7-3                                      amd64        FreeSWITCH Sounds
ii  bbb-fsesl-akka                        1:2.7-28                                     all          BigBlueButton FS-ESL (Akka)
ii  bbb-html5                             1:2.7-243                                    amd64        The HTML5 components for BigBlueButton
ii  bbb-html5-nodejs                      1:2.7-6                                      amd64        Include a specific NodeJS version for bbb-html5
ii  bbb-learning-dashboard                1:2.7-13                                     amd64        BigBlueButton bbb-learning-dashboard
ii  bbb-libreoffice-docker                1:2.7-7                                      amd64        BigBlueButton setup for LibreOffice running in docker
ii  bbb-mkclean                           1:2.7-3                                      amd64        Clean and optimize Matroska and WebM files
ii  bbb-pads                              1:2.7-12                                     amd64        BigBlueButton Pads
ii  bbb-playback                          1:2.7-7                                      amd64        Player for BigBlueButton presentation format recordings
ii  bbb-playback-presentation             1:2.7-5                                      amd64        BigBluebutton playback of presentation
ii  bbb-record-core                       1:2.7-13                                     amd64        BigBlueButton record and playback
ii  bbb-web                               1:2.7-47                                     amd64        BigBlueButton API
ii  bbb-webrtc-recorder                   1:2.7-7                                      amd64        BigBlueButton WebRTC Recorder
ii  bbb-webrtc-sfu                        1:2.7-33                                     amd64        BigBlueButton WebRTC SFU
ii  bigbluebutton                         1:2.7-2                                      amd64        Open source web conferencing platform (bbb)

isn't there a problem with the BBB version ?

[farhatahmad]

Nope, this is unrelated to BBB. One final thing, maybe this will work

docker exec -it greenlight-v3 bundle exec rails c
Tenant.create(name: "greenlight", client_secret: 'test')

Try that and then re-run the commands

[zulf10]

Return :

docker exec -it greenlight-v3 bundle exec rails c
Loading production environment (Rails 7.1.3.1)
irb(main):001> Tenant.create(name: "greenlight", client_secret: 'test')
=> 
#<Tenant:0x00007f786213e2a0
 name: "greenlight",
 client_secret: "[FILTERED]",
 created_at: Wed, 13 Mar 2024 18:31:08.914868000 UTC +00:00,
 updated_at: Wed, 13 Mar 2024 18:31:08.914868000 UTC +00:00,
 id: "b40fe1e9-fb8a-4467-8661-8e981a8abb1b">

And run again commands migrate role do same things:

# docker exec -it greenlight-v2 bundle exec rake migrations:roles
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:68: warning: already initialized constant Net::ProtocRetryError
/usr/local/lib/ruby/2.7.0/net/protocol.rb:66: warning: previous definition of ProtocRetryError was here
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:208: warning: already initialized constant Net::BufferedIO::BUFSIZE
/usr/local/lib/ruby/2.7.0/net/protocol.rb:206: warning: previous definition of BUFSIZE was here
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:504: warning: already initialized constant Net::NetPrivate::Socket
/usr/local/lib/ruby/2.7.0/net/protocol.rb:503: warning: previous definition of Socket was here
warning: parser/current is loading parser/ruby27, which recognizes2.7.6-compliant syntax, but you are running 2.7.7.
Please see https://github.com/whitequark/parser#compatibility-with-ruby-mri.
Unable to migrate Role:
ID: 6
Name: Moderateur
Errors: Provider does not exist
Unable to migrate Role:
ID: 7
Name: Basic
Errors: Provider does not exist
Unable to migrate Role:
ID: 8
Name: Membre niv. 2
Errors: Provider does not exist

Roles migration complete.
In case of an error please retry the process to resolve.

[farhatahmad]

Oh - I just looked through your env variables

V3_ENDPOINT=https://XXX/bigbluebutton/ 

This should be pointing at wherever your v3 server is deployed, not at your BigBlueButton server. How did you install v3?

[zulf10]

BigBlueButton server, v2 and v3 are on same server.

Web interface v3 on https://XXX/ Web interface v2 on https://XXX/b

But endpoint already same on v2 and v3 env

[psy]

Web interface v3 on https://XXX/

try V3_ENDPOINT=https://XXX/

[zulf10]

Same issue with V3_ENDPOINT=https://XXX/

do the containers have to be on the same docker network ?

[psy]

No, v2 only needs to be able to reach the url specified via V3_ENDPOINT. I just recently migrated a greenlight installation to a completely different server. Do you run both versions under the same domain (and use greenlights nginx config)? If yes, your api routes mights be forwarded to the old instance.

Try curl -H 'Accept: application/json' https://XXX/api/v1/env.json and look for VERSION_TAG.

[zulf10]

Yes both versions run under the same domain and use greenlights nginx config.

Command curl -H 'Accept: application/json' https://XXX/api/v1/env.json return "VERSION_TAG":"release-3.3.1"

[psy]

lgtm. i'm out of ideas.

[caos30]

Hello, I've been suffering from the "same" problem, and I have read on other threads that this issue is at least 1 year old. In my case, I have the Greenlight v3 running in a different VM than the v2. I followed the same instructions and checks as above with similar results.

Indeed, nobody above commented on this, but I suppose that the OP did it: put the same SECRET KEY in both .env files, for SECRET_KEY_BASE (v2 & v3) and for V3_SECRET_KEY_BASE (v2).

In my case, the return of the command:

sudo docker exec -it greenlight-v2 bundle exec rake migrations:roles

is this:

/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:68: warning: already initialized constant Net::ProtocRetryError
/usr/local/lib/ruby/2.7.0/net/protocol.rb:66: warning: previous definition of ProtocRetryError was here
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:208: warning: already initialized constant Net::BufferedIO::BUFSIZE
/usr/local/lib/ruby/2.7.0/net/protocol.rb:206: warning: previous definition of BUFSIZE was here
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:504: warning: already initialized constant Net::NetPrivate::Socket
/usr/local/lib/ruby/2.7.0/net/protocol.rb:503: warning: previous definition of Socket was here
warning: parser/current is loading parser/ruby27, which recognizes2.7.6-compliant syntax, but you are running 2.7.7.
Please see https://github.com/whitequark/parser#compatibility-with-ruby-mri.

Roles migration complete.

I think the message "Roles migration complete" is wrong, isn't it? I think that if the migrations were successful, I should see a list of the roles migrated. Am I wrong?

When i run the next command (to migrate users):

sudo docker exec -it greenlight-v2 bundle exec rake migrations:users

then i get this:

/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:68: warning: already initialized constant Net::ProtocRetryError
/usr/local/lib/ruby/2.7.0/net/protocol.rb:66: warning: previous definition of ProtocRetryError was here
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:208: warning: already initialized constant Net::BufferedIO::BUFSIZE
/usr/local/lib/ruby/2.7.0/net/protocol.rb:206: warning: previous definition of BUFSIZE was here
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:504: warning: already initialized constant Net::NetPrivate::Socket
/usr/local/lib/ruby/2.7.0/net/protocol.rb:503: warning: previous definition of Socket was here
warning: parser/current is loading parser/ruby27, which recognizes2.7.6-compliant syntax, but you are running 2.7.7.
Please see https://github.com/whitequark/parser#compatibility-with-ruby-mri.

Unable to migrate User:
UID: gl-mzyagccgkozf
Name: personal_user_name_1
Provider: greenlight
Errors: ["InvalidParams"]

Unable to migrate User:
UID: gl-wymjnjolxued
Name: personal_user_name_2
Provider: greenlight
Errors: ["InvalidParams"]

Unable to migrate User:
UID: gl-bnbdsokhbqbl
Name: personal_user_name_3
Provider: greenlight
Errors: ["InvalidParams"]

Unable to migrate User:
UID: gl-kctsrjtrpejx
Name: personal_user_name_4
Provider: greenlight
Errors: ["InvalidParams"]

Unable to migrate User:
UID: gl-elkertszzprw
Name: personal_user_name_5
Provider: greenlight
Errors: ["InvalidParams"]

Unable to migrate User:
UID: gl-jvwfwvxiusmp
Name: personal_user_name_6
Provider: greenlight
Errors: ["InvalidParams"]

Users migration completed.
In case of an error please retry the process to resolve.
If you have not migrated your roles, kindly run 'rake migrations:roles' first and then retry.

So, I understand that the migrations didn't complete, and I suppose that this is the cause of the "unable to migrate user" error messages.

I'm really not sure about what the expected result of each command is because this is my first migration from v2 to v3. Although the official guide for it (https://docs.bigbluebutton.org/greenlight/v3/migration/) mentions the results for a successful migration, it doesn't cover a "strange mid-successful migration". I'm not very clear on how to understand these messages.

So, any help will be very welcomed.

[caos30]

Here's the grammatically corrected version:

Corrected Version

Please, can at least someone guide me on a way to migrate recordings done in v2 to the v3 server MANUALLY and enable users on v3 to manage and share those recordings from Greenlight v3?

I've manually recreated the roles and users of v2 in v3. But we need now to copy the recordings of the last 2 years from v2 to the v3 server and make them accessible for users on v3. How can I do that? Taking into account that the official "migration script" is failing and nobody is able to help us make it run. 🤗

Any help is welcomed. Thanks in advance.

Note: just after install the v3 server i set the same SECRET_KEY (editing greenligh-v3/.env file) existing at greenlight/.env file on v2 server. I suspect that this key would be important to some "crypted file names".

[zulf10]

Hello,

We migrated again. We removed the roles. We had to modify the script migrations.rake in /lib/tasks/migrations

We added instructions to the script like this:

@@ -68,7 +68,7 @@
         .where.not(roles: { name: COMMON[:filtered_user_roles] }, deleted: true)
         .find_each(start: start, finish: stop, batch_size: COMMON[:batch_size]) do |u|
       role_name = infer_role_name(u.role.name)
-      params = { user: { name: u.name, email: u.email, external_id: u.social_uid, language: u.language, role: role_name } }
+      params = { user: { name: u.name, email: u.email, external_id: u.social_uid, language: u.language, role: role_name, provider: 'greenlight' } }

       response = Net::HTTP.post(uri('users'), payload(params), COMMON[:headers])

@@ -133,7 +133,8 @@
                          last_session: r.last_session&.to_datetime,
                          owner_email: r.owner.email,
                          room_settings: room_settings,
-                         shared_users_emails: shared_users_emails } }
+                         shared_users_emails: shared_users_emails,
+                         provider: 'greenlight' } }

       response = Net::HTTP.post(uri('rooms'), payload(params), COMMON[:headers])

@@ -188,7 +189,7 @@
       glRequireAuthentication: infer_room_config_value(setting.get_value('Room Authentication'))
     }.compact

-    params = { settings: { site_settings: site_settings, room_configurations: room_configurations } }
+    params = { settings: { site_settings: site_settings, room_configurations: room_configurations, provider: 'greenlight' } }

     response = Net::HTTP.post(uri('settings'), payload(params), COMMON[:headers])

We didn't migrate settings or records

We check again but it seem solved for us

[CrossFire10100]

Same issue here.. I wonder why they changed the whole DB structure making it virtually impossible to migrate correctly any data at all.

If anyone wants to migrate recordings just create the old rooms with the same room name and add to the docker compose file pgadmin on both sides login to both postgres db and just clone the old meeting id to the new room and resync recordings.

[herzkerl]

We're also seeing failed user migrations with the error Provider does not exist—I've checked everything in this thread and can't find any issues within the .env files.

The solution provided by @zulf10 worked in this case—thanks!