search

sympa-community / sympa

Sympa, Mailing List Management Software
https://www.sympa.community/sympa
GNU General Public License v2.0
241 stars 95 forks source link

List used as a Data Source cannot be deleted despite no longer a DS for any list #1660

Closed bwiderski-HBS closed 8 months ago

bwiderski-HBS commented 1 year ago

List it_privileged_owners was used as an Owner Data Source for multiple other lists but has been superseded so needs to be closed.

After removing it as a DS from all lists, plus manually synchronizing those lists' owners, I still cannot close it because its admin page still says it is included in other lists.

Pressing the button "View lists including it_privileged_owners" takes me to https://lists.hbs.edu/lists/including_lists/it_privileged_owners which only displays a blue box saying "No Lists".

I restarted sympa and wwsympa - no change. 15 hours after first seeing this behavior it still exists.

Version 6.2.70 Screen shots attached.

Screenshots

sympa 01 sympa 02

bwiderski-HBS commented 1 year ago

FYI I just had a second occurrence - steps to have happen:

  1. Went to Admin of tss_privileged_owners - said it was included in one list: mba_auth
  2. Went to mba_auth, edited owner data source by a. changed data source name from "tss_privileged_owners" to "tier_2_privileged_owners" b. Went to Owners page and synchronized data sources: old tss_privileged_owners content was removed, new tier_2_privileged_owners content added.
  3. Returned to tss_privileged_owners Admin - still said it was included in another list.
  4. Clicked button to see where included - again the result was a blue box stating "no lists"
ikedas commented 1 year ago

@bwiderski-HBS , what is the database you are using?

ikedas commented 1 year ago

@bwiderski-HBS , additionally, could you please show the system log during the steps above?

bwiderski-HBS commented 1 year ago

Postgres locally on the same server.

Will look for log info on Monday when I’m back at the office (EDT)

Betty

On Apr 21, 2023, at 7:45 PM, IKEDA Soji @.***> wrote:

 You don't often get email from @.*** Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification

@bwiderski-HBShttps://github.com/bwiderski-HBS , what is the database you are using?

— Reply to this email directly, view it on GitHubhttps://github.com/sympa-community/sympa/issues/1660#issuecomment-1518426340, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A5PL3EMPD36ZK2HERI5GG23XCMLYTANCNFSM6AAAAAAXG54AMU. You are receiving this because you were mentioned.Message ID: @.***>

bwiderski-HBS commented 1 year ago

042123 syslogs re sympa.txt Attached is the system log segment from the period of my Sympa activity the afternoon of Friday April 21. During that period I was doing a number of things:

After removing or renaming tss_privileged_owners as a data source from all other lists I wanted to close it - but Sympa still thinks it is in use by other lists, but when asked to identify those it gives the blue box "no lists".

ikedas commented 1 year ago

Hi @bwiderski-HBS ,

Could you show us the inclusion settings of the list "mba2024i>@</ilists.hbs.edu"?

The log tells that this list is failing inclusion.

Apr 21 13:58:15 rhlists2 task_manager[680148]: err main::#148 > Sympa::Spindle::spin#83 > Sympa::Spindle::ProcessTask::_twist#74 > Sympa::Spindle::ProcessTask::_execute#140 > Sympa::Spindle::ProcessTask::_cmd_process#174 > Sympa::Spindle::ProcessTask::do_sync_include#1347 > Sympa::List::sync_include#4107 > Sympa::Spindle::spin#95 > Sympa::Request::Handler::include::_twist#209 > Sympa::Request::Handler::include::_update_users#377 > Sympa::DataSource::open#113 > Sympa::DataSource::List::_open#60 > Sympa::List::new#137 Missing robot parameter, cannot create list object for mba2024b
bwiderski-HBS commented 1 year ago

Attached PDF of mba2024 data sources setup.

From: IKEDA Soji @.> Sent: Tuesday, April 25, 2023 10:12 PM To: sympa-community/sympa @.> Cc: Widerski, Betty @.>; Mention @.> Subject: Re: [sympa-community/sympa] List used as a Data Source cannot be deleted despite no longer a DS for any list (Issue #1660)

You don't often get email from @.**@.>. Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification

Hi @bwiderski-HBShttps://github.com/bwiderski-HBS ,

Could you show us the inclusion settings of the list @.**@.>"?

- Reply to this email directly, view it on GitHubhttps://github.com/sympa-community/sympa/issues/1660#issuecomment-1522662498, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A5PL3EJRZYDIWXMMI6I2XIDXDCAAPANCNFSM6AAAAAAXG54AMU. You are receiving this because you were mentioned.Message ID: @.**@.>>

ikedas commented 1 year ago

@bwiderski-HBS sorry, attachment in the e-mail reply is removed. Please post it on the web.

bwiderski-HBS commented 1 year ago

mba2024 - inclusions.pdf

ikedas commented 1 year ago

Thanks.

The log above tells that the list "mba2024b" does no longer exist.

  1. Open the data source config page of "mba2024" list.
  2. Check the "Delete" checkbox at the right side of the data source (source) "local_list (local_list)" for "mba2024b", then click "Apply modifications" at the bottom of the page.
  3. Wait for a whle (one hour or so).
  4. Check the log and confirm that the err log about "sync_include" is no longer shown.

Then check whether "it_privileged_owners" may be closed.

bwiderski-HBS commented 1 year ago

Will do - will also check if the same problem with tss_privileged_owners resolves.

From: IKEDA Soji @.> Sent: Wednesday, April 26, 2023 10:02 AM To: sympa-community/sympa @.> Cc: Widerski, Betty @.>; Mention @.> Subject: Re: [sympa-community/sympa] List used as a Data Source cannot be deleted despite no longer a DS for any list (Issue #1660)

You don't often get email from @.**@.>. Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification

Thanks.

The log above tells that the list "mba2024b" does no longer exist.

  1. Open the data source config page of "mba2024" list.
  2. Check the "Delete" checkbox at the right side of the data source (source) "local_list (local_list)" for "mba2024b", then click "Apply modifications" at the bottom of the page.
  3. Wait for a whle (one hour or so).
  4. Check the log and confirm that the err log about "sync_include" is no longer shown.

Then check whether "it_privileged_owners" may be closed.

- Reply to this email directly, view it on GitHubhttps://github.com/sympa-community/sympa/issues/1660#issuecomment-1523475634, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A5PL3EPQRBMI7BX676UE5MTXDETELANCNFSM6AAAAAAXG54AMU. You are receiving this because you were mentioned.Message ID: @.**@.>>

bwiderski-HBS commented 1 year ago

042623 syslog re sympa.txt

Today's logs. Still cannot delete it_privileged_owners or tss_privileged_owners.

Note - I'm OOO the rest of this week so will respond to anything 1 May.

ikedas commented 1 year ago

You may have to log-in as a listmaster and check the "View lists including it_privileged_owners".

Perhaps the "Visibility of the list (visibility)" setting of these lists prevent them from being displayed. Listmasters can work around this limitation in most cases.

ikedas commented 1 year ago

Oh you are the listmaster @bwiderski-HBS .

Can you apply following modification on web template?

--- a/default/web_tt2/including_lists.tt2
+++ b/default/web_tt2/including_lists.tt2
@@ -8,7 +8,7 @@
         <span class="divider"></span>
         [% SET invisible_count = 0; visible_count = 0 ~%]
         [% FOREACH l = which ~%]
-            [% IF l.value.visible ~%]
+            [% IF l.value.visible || is_listmaster ~%]
                 <div class="item_content">
                     <a class="item_title" href="[% l.value.robot == robot ? l.value.url_rel : l.value.url_abs %]">
                         [%|obfuscate(conf.spam_protection) ~%]
bwiderski-HBS commented 1 year ago

I’m not understanding which file you mean by “web template” and your syntax- by

a/default/web_tt2/including_lists.tt2 +++ b/default/web_tt2/including_lists.tt2 @@ -8,7 +8,7 @@

Do you mean to edit the file including_lists.tt2 in the default web_tt2 directory and add the lines here that are written below “@@ -8,7 +8,7 @@“ ?

Is there any risk? This is my prod server - I did not have this issue on my dev server.

Betty

On Apr 27, 2023, at 4:52 AM, IKEDA Soji @.***> wrote:



Oh you are the listmaster @bwiderski-HBShttps://github.com/bwiderski-HBS .

Can you apply following modification on web template?

--- a/default/web_tt2/including_lists.tt2 +++ b/default/web_tt2/including_lists.tt2 @@ -8,7 +8,7 @@ [% SET invisible_count = 0; visible_count = 0 ~%] [% FOREACH l = which ~%]

— Reply to this email directly, view it on GitHubhttps://github.com/sympa-community/sympa/issues/1660#issuecomment-1525168527, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A5PL3EJZP7YOYDLML6QQB7DXDIXV7ANCNFSM6AAAAAAXG54AMU. You are receiving this because you were mentioned.Message ID: @.***>

ikedas commented 1 year ago

Hi @bwiderski-HBS ,

Please check the manual to know how to customize template.

The colored texts mean that, you create the template file web_tt2/including_list.tt2 replacing the line

            [% IF l.value.visible ~%]

with

            [% IF l.value.visible || is_listmaster ~%]

This will not break Sympa.

ikedas commented 1 year ago

Hi @bwiderski-HBS again,

Apr 21 14:25:02 rhlists2 wwsympa[680421]: err main::#1557 > main::do_create_list#9515 > Sympa::Spindle::spin#95 > Sympa::Request::Handler::create_list::_twist#215 > Sympa::List::new#181 > Sympa::List::load#699 > Sympa::List::_load_list_config_file#5044 Unknown key "visibility" in paragraph "available_user_options" in /var/lib/sympa/list_data/lists.hbs.edu/test01-post-edit/config

The config of the list "test01-post-edit" seems broken. Maybe any create list template is broken and the lists created using it is/are also broken.

Please check the log and find out the other broken lists, and if these lists are unnecessary, please close them.

bwiderski-HBS commented 1 year ago

Yes - on Monday when I’m back in the office.

Betty

On Apr 28, 2023, at 2:27 AM, IKEDA Soji @.***> wrote:



Hi @bwiderski-HBShttps://github.com/bwiderski-HBS again,

Apr 21 14:25:02 rhlists2 wwsympa[680421]: err main::#1557 > main::do_create_list#9515 > Sympa::Spindle::spin#95 > Sympa::Request::Handler::create_list::_twist#215 > Sympa::List::new#181 > Sympa::List::load#699 > Sympa::List::_load_list_config_file#5044 Unknown key "visibility" in paragraph "available_user_options" in /var/lib/sympa/list_data/lists.hbs.edu/test01-post-edit/config

The config of the list "test01-post-edit" seems broken. Maybe any create list template is broken and the lists created using it is also broken.

Please check the log and find out the other broken lists, and if these lists are unnecessary, please close them.

— Reply to this email directly, view it on GitHubhttps://github.com/sympa-community/sympa/issues/1660#issuecomment-1527054915, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A5PL3EIG5NI7FWSXH63IC3LXDNPOTANCNFSM6AAAAAAXG54AMU. You are receiving this because you were mentioned.Message ID: @.***>

ikedas commented 1 year ago

I’ll be on vacation in the next week, so I’ll be less responsive.

ikedas commented 1 year ago

@bwiderski-HBS , additionally, please also show us the log for a few minutes including the time when you clicked on the "Admin" link.

bwiderski-HBS commented 1 year ago

Replies to your various comments while I was OOO:

my templates: send no_post_bounce

available_user_options
reception mail
visibility conceal

Default Sympa template html-news-letter:

visibility noconceal

send editorkeyonly

available_user_options 
reception mail,nomail,txt,html

Thus all my lists have "visibility noconceal" since the visibility parameter wasn't seen properly. This error only appeared in logs during list creation, so it seems there is nothing "broken" about the created lists, just that I need to manually set their visibility to "conceal" and fix my templates for new creations.

This doesn't seem to be related to my data sources problem, as the templates are the same on my dev server which does not have the DS problem. pre-including_list-tt2-change.txt post-including_list-tt2-edit-and restart.txt

ikedas commented 1 year ago

Hi @bwiderski-HBS ,

Replies to your various comments while I was OOO:

  • Sorry, I was only looking at the email fo your comment, not the github entry, so I wasn't seeing the formatting, including color.

I understand. From now on I'll write without assuming the web.

  • I made the change in default/web-tt2/including_lists.tt2 and restarted sympa and wwsympa. I am attaching syslogs from 1) around using admin before doing the including_lists edit, and 2) after doing it and restarting. I'm seeing no difference in behavior.

These are the logs that I asked for showing us 'the log for a few minutes including the time when you clicked on the "Admin" link'.

Next, please show us the log from when you clicked on the "Admin" link' to when you clicked on the "View lists including ..." and see the text "No lists".

  • Looks like ALL my lists had that 'Unknown key "visibility" in paragraph "available_user_options' on creation. I've been using templates of a format created by the consultant we hired to set up Sympa. All of the templates put "visibility conceal" in a location that makes Sympa read it as a variable pertaining to available_user_options:

my templates: send no_post_bounce

available_user_options
reception mail
visibility conceal

This shiould be as below:

available_user_options
reception mail

visibility conceal

Thus all my lists have "visibility noconceal" since the visibility parameter wasn't seen properly. This error only appeared in logs during list creation, so it seems there is nothing "broken" about the created lists, just that I need to manually set their visibility to "conceal" and fix my templates for new creations.

This doesn't seem to be related to my data sources problem, as the templates are the same on my dev server which does not have the DS problem.

This may not be directly related to the "No lists" issue, but it is not working correctly and should be fixed.

bwiderski-HBS commented 1 year ago

Yes, I've fixed the incorrect templates.

Log:

[sympa@rhlists2 sympa]$ date Wed May 3 08:56:17 EDT 2023 [sympa@rhlists2 sympa]$ tail -f /var/log/messages May 3 08:55:06 rhlists2 systemd[1]: pmlogger_check.service: Succeeded. May 3 08:55:10 rhlists2 systemd[1]: Starting Check and migrate non-primary pmlogger farm instances... May 3 08:55:10 rhlists2 systemd[1]: Started Check and migrate non-primary pmlogger farm instances. May 3 08:55:10 rhlists2 systemd[1]: pmlogger_farm_check.service: Succeeded. May 3 08:55:43 rhlists2 wwsympa[1849747]: info main::do_lists(, ) [robot lists.hbs.edu] [session 60721619194124] [client 192.168.3.1] [user bwiderski@hbs.edu] May 3 08:55:43 rhlists2 wwsympa[1849747]: info Sympa::WWW::Session::renew() [robot lists.hbs.edu] [session 60721619194124] [client 192.168.3.1] [user bwiderski@hbs.edu] new session 68812186092189 May 3 08:56:00 rhlists2 systemd[1]: Starting system activity accounting tool... May 3 08:56:00 rhlists2 systemd[1]: sysstat-collect.service: Succeeded. May 3 08:56:00 rhlists2 systemd[1]: Started system activity accounting tool. May 3 08:56:19 rhlists2 rsyslogd[1561]: -- MARK -- May 3 08:56:37 rhlists2 wwsympa[1849745]: info main::do_info() [robot lists.hbs.edu] [session 68812186092189] [client 192.168.3.1] [user bwiderski@hbs.edu] [list it_privileged_owners] May 3 08:56:37 rhlists2 wwsympa[1849745]: info Sympa::WWW::Session::renew() [robot lists.hbs.edu] [session 68812186092189] [client 192.168.3.1] [user bwiderski@hbs.edu] new session 25733895167847 May 3 08:56:40 rhlists2 wwsympa[1849746]: info main::do_admin() [robot lists.hbs.edu] [session 25733895167847] [client 192.168.3.1] [user bwiderski@hbs.edu] [list it_privileged_owners]

ikedas commented 1 year ago

Could you please show the log after you clicked the "Admin" link?

bwiderski-HBS commented 1 year ago

I did - after I did "Date" I went to the UI, went to the list, clicked Admin, then the view button - this is the entire log from doing those steps immediately after 08:56:17. This is /var/log/messages - is there a different log you want to see?

ikedas commented 1 year ago

Did you see "No lists"?

bwiderski-HBS commented 1 year ago

Correct - still gave "no lists" when hitting the "view lists using" button.

From: IKEDA Soji @.> Sent: Wednesday, May 03, 2023 11:12 PM To: sympa-community/sympa @.> Cc: Widerski, Betty @.>; Mention @.> Subject: Re: [sympa-community/sympa] List used as a Data Source cannot be deleted despite no longer a DS for any list (Issue #1660)

Did you see "No lists"?

- Reply to this email directly, view it on GitHubhttps://github.com/sympa-community/sympa/issues/1660#issuecomment-1534028218, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A5PL3EMQ5KQINDNQ3Z2VWSDXEMM77ANCNFSM6AAAAAAXG54AMU. You are receiving this because you were mentioned.Message ID: @.**@.>>

ikedas commented 1 year ago

Can you use psql command? If you can, could you connect to database, run following query and show the result?

SELECT * FROM inclusion_table WHERE source_inclusion = 'it_privileged_owners@lists.hbs.edu';

And, if the result is not empty, do the list names included in it really exist?

bwiderski-HBS commented 1 year ago

I did this for both it_privileged_owners and tss_privileged_owners. The results for both were not empty, and none of the lists are either active or in "closed".

owner it_privileged_owners@lists.hbs.edu tss-closed-open-dharvill@lists.hbs.edu 1680100563 owner it_privileged_owners@lists.hbs.edu testopenpost@lists.hbs.edu 1680104315 owner it_privileged_owners@lists.hbs.edu testopenpost2@lists.hbs.edu 1680105162 owner it_privileged_owners@lists.hbs.edu test-tss-closed-closed-bw@lists.hbs.edu 1680099594 owner it_privileged_owners@lists.hbs.edu testbrian@lists.hbs.edu 1678467125

owner tss_privileged_owners@lists.hbs.edu tss-closed-open-dharvill@lists.hbs.edu 1680100563 owner tss_privileged_owners@lists.hbs.edu bwmigrationtest_auth@lists.hbs.edu 1648671518 owner tss_privileged_owners@lists.hbs.edu testbrian@lists.hbs.edu 1678467125 owner tss_privileged_owners@lists.hbs.edu test-tss-closed-closed-bw@lists.hbs.edu 1680099594 owner tss_privileged_owners@lists.hbs.edu bwmigrationtest@lists.hbs.edu 1648659218

I have full Postgres database nightly dumps, if seeing one of those would be helpful.

ikedas commented 1 year ago

So those lists do not really exist. Then I don't understand why nothing is showing up in the logs.... Anyway, could you please apply this patch and restart the services to see if it solves the problem?

bwiderski-HBS commented 1 year ago

Please advise how to "apply this patch" - I see it is changes to /usr/share/sympa/lib/List.pm on my server, but I do not know how to get a copy of the modified List.pm to replace existing. We do not have our Sympa components in Git, and I'm hoping there's an easier way than my manually viming and editing by hand?

ikedas commented 1 year ago

You may use patch(1) command to apply the patch I presented. For more details to use the command please ask your colleagues who have knowleges about software development. Or, please wait for a new release of Sympa in the future which includes this patch.

bwiderski-HBS commented 1 year ago

I have figured out how to use patch(1) to update List.pm. I am waiting for a scheduled patching window to apply it long enough to remove the problem lists, then will probably revert to the current file version until you issue a new release, since this is our prod instance.

ikedas commented 10 months ago

The patch will be included in the next release. At that time, this issue will be closed.