ctlcltd / e2-sat-editor

Satellite channel lists editor with tabbed nav 📡 for Enigma2, Neutrino, Lamedb, dreambox lists
https://e2sateditor.com
GNU General Public License v3.0
16 stars 1 forks source link

FTP and parser errors with OpenVix #24

Closed harmlessdrudge closed 2 months ago

harmlessdrudge commented 2 months ago

Describe the bug

My first time running this (on Linux Mint 21.3), using OpenVix 6.4. on a ZGemma H7S: [UI looks good]

I changed the picon path to /media/hdd/picon

I don't recall where I found advice on installing picons but I did it with FTP not using an extension (didn't know it was an option).

Downloading from my system I get the following errors

Errors during FTP operations (e2SAT editor and Openix 6.4)

Parser Error: userbouquet (bouquets.tv:5) [parse_e2db_bouquet] (i=0)
Parser Error: userbouquet (bouquets.tv:23) [parse_e2db_bouquet] (i=1)
Parser Error: userbouquet (bouquets.tv:41) [parse_e2db_bouquet] (i=2)
Parser Error: userbouquet (bouquets.radio:5) [parse_e2db_bouquet] (i=3)

FTP Error: Remote file not found [download_data] (/etc/enigma2/AutoBouquetsMaker)
FTP Error: Remote file not found [download_data] (/etc/enigma2/crossepg)
FTP Error: Remote file not found [download_data] (/etc/tuxbox/config)
FTP Error: Remote file not found [download_data] (/etc/tuxbox/scce)

I don't know the significance of the first 4 yet, but the next 4 seem wrong or misleading. The "files" listed are directories and they do exist. What am I missing?

Steps to reproduce

  1. ... Make an FTP download from OpenVix 6.4

Expected behavior

No FTP errors (I am reluctant to write data back to the box with errors on download).

I'd like to know what to do to -- if necessary -- revert to a default set of bouquets.

Software version

1.5

Download medium

GitHub Releases page

Platform running

I don't know

Architecture running

I don't know

Other (please specify)

Desktop environment is Cinnamon.

DVB-S/2 source is Astra 28.2E; DVB-T sources is Saorview

Additional context

FTP download

Contents of /etc/tuxbox

-rw-r--r--    1 root     root         14433 Jan  6  2024 atsc.xml
-rw-r--r--    1 root     root        801551 Jan  6  2024 cables.xml
drwxr-xr-x    2 root     root          4096 Oct 10  2020 config
-rw-r--r--    1 root     root        481514 Jan  6  2024 satellites.xml
drwxr-xr-x    2 root     root          4096 Sep 18  2023 scce
-rw-r--r--    1 root     root        360796 Jan  6  2024 terrestrial.xml
-rw-r--r--    1 root     root         29415 Jan  6  2024 timezone.xml

Contents of bouquets.tv

`#NAME Bouquets (TV)

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.terrestrial_ie_saorview_PSB1.main.tv" ORDER BY bouquet

SERVICE 1:519:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.terrestrial_ie_saorview_PSB1.separator.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.terrestrial_ie_saorview_PSB1.hd.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.terrestrial_ie_saorview_PSB1.ftahd.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.terrestrial_ie_saorview_PSB1.fta.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.main.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.101.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.200.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.300.tv" ORDER BY bouquet

SERVICE 1:519:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.400.tv" ORDER BY bouquet

SERVICE 1:519:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.500.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.600.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.650.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.700.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.800.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.949.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.970.tv" ORDER BY bouquet

SERVICE 1:519:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.998.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.999.tv" ORDER BY bouquet

SERVICE 1:519:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.separator.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.hd.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.woodlandstv_tv.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de00.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de01.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de02.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de03.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de04.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de05.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de06.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de07.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de08.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de09.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de10.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de11.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de12.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de13.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de14.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de15.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.LastScanned.tv" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.bskyb_tv.tv" ORDER BY bouquet

`

Contents of bouquets.radio

`

NAME Bouquets (Radio)

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.terrestrial_ie_saorview_PSB1.main.radio" ORDER BY bouquet

SERVICE 1:519:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.main.radio" ORDER BY bouquet

SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de00.radio" ORDER BY bouquet

SERVICE 1:519:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de01.radio" ORDER BY bouquet

SERVICE 1:519:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de02.radio" ORDER BY bouquet

`

Bouquets (*.tv)

bouquets.tv userbouquet.LastScanned.tv userbouquet.abm.sat_282_freesat.101.tv userbouquet.abm.sat_282_freesat.200.tv userbouquet.abm.sat_282_freesat.300.tv userbouquet.abm.sat_282_freesat.400.tv userbouquet.abm.sat_282_freesat.500.tv userbouquet.abm.sat_282_freesat.600.tv userbouquet.abm.sat_282_freesat.650.tv userbouquet.abm.sat_282_freesat.700.tv userbouquet.abm.sat_282_freesat.800.tv userbouquet.abm.sat_282_freesat.949.tv userbouquet.abm.sat_282_freesat.970.tv userbouquet.abm.sat_282_freesat.998.tv userbouquet.abm.sat_282_freesat.999.tv userbouquet.abm.sat_282_freesat.hd.tv userbouquet.abm.sat_282_freesat.main.tv userbouquet.abm.sat_282_freesat.separator.tv userbouquet.abm.terrestrial_ie_saorview_PSB1.fta.tv userbouquet.abm.terrestrial_ie_saorview_PSB1.ftahd.tv userbouquet.abm.terrestrial_ie_saorview_PSB1.hd.tv userbouquet.abm.terrestrial_ie_saorview_PSB1.main.tv userbouquet.abm.terrestrial_ie_saorview_PSB1.separator.tv userbouquet.bskyb__tv_.tv userbouquet.de00.tv userbouquet.de01.tv userbouquet.de02.tv userbouquet.de03.tv userbouquet.de04.tv userbouquet.de05.tv userbouquet.de06.tv userbouquet.de07.tv userbouquet.de08.tv userbouquet.de09.tv userbouquet.de10.tv userbouquet.de11.tv userbouquet.de12.tv userbouquet.de13.tv userbouquet.de14.tv userbouquet.de15.tv userbouquet.woodlandstv__tv_.tv Bouquets (*.radio)

bouquets.radio userbouquet.abm.sat_282_freesat.main.radio userbouquet.abm.terrestrial_ie_saorview_PSB1.main.radio userbouquet.de00.radio userbouquet.de01.radio userbouquet.de02.radio

Log

2024-07-15 16:03:24.417411 <Debug> [e2se.gui] e2db::parse_e2db_bouquet  filename: bouquets.tv
2024-07-15 16:03:24.417481 <Error> [e2se.gui] e2db::parse_e2db_bouquet  Parser Error: userbouquet (bouquets.tv:5)
2024-07-15 16:03:24.417529 <Error> [e2se.gui] e2db::parse_e2db_bouquet  Parser Error: userbouquet (bouquets.tv:23)
2024-07-15 16:03:24.417567 <Error> [e2se.gui] e2db::parse_e2db_bouquet  Parser Error: userbouquet (bouquets.tv:41)
2024-07-15 16:03:24.417631 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.abm.terrestrial_ie_saorview_PSB1.main.tv
2024-07-15 16:03:24.417792 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.abm.terrestrial_ie_saorview_PSB1.hd.tv
2024-07-15 16:03:24.419687 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.abm.terrestrial_ie_saorview_PSB1.ftahd.tv
2024-07-15 16:03:24.421232 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.abm.terrestrial_ie_saorview_PSB1.fta.tv
2024-07-15 16:03:24.422871 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.abm.sat_282_freesat.main.tv
2024-07-15 16:03:24.424401 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.abm.sat_282_freesat.101.tv
2024-07-15 16:03:24.424515 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.abm.sat_282_freesat.200.tv
2024-07-15 16:03:24.424545 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.abm.sat_282_freesat.300.tv
2024-07-15 16:03:24.424562 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: ���
2024-07-15 16:03:24.424572 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.abm.sat_282_freesat.600.tv
2024-07-15 16:03:24.424586 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.abm.sat_282_freesat.650.tv
2024-07-15 16:03:24.424607 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.abm.sat_282_freesat.700.tv
2024-07-15 16:03:24.424672 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.abm.sat_282_freesat.800.tv
2024-07-15 16:03:24.424699 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.abm.sat_282_freesat.949.tv
2024-07-15 16:03:24.424738 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.abm.sat_282_freesat.970.tv
2024-07-15 16:03:24.424752 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: ���
2024-07-15 16:03:24.424761 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.abm.sat_282_freesat.999.tv
2024-07-15 16:03:24.424809 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.abm.sat_282_freesat.hd.tv
2024-07-15 16:03:24.426333 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.woodlandstv__tv_.tv
2024-07-15 16:03:24.426408 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.de00.tv
2024-07-15 16:03:24.426584 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.de01.tv
2024-07-15 16:03:24.426618 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.de02.tv
2024-07-15 16:03:24.426793 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.de03.tv
2024-07-15 16:03:24.426863 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.de04.tv
2024-07-15 16:03:24.426984 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.de05.tv
2024-07-15 16:03:24.427009 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.de06.tv
2024-07-15 16:03:24.427022 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.de07.tv
2024-07-15 16:03:24.427051 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.de08.tv
2024-07-15 16:03:24.427072 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.de09.tv
2024-07-15 16:03:24.427155 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.de10.tv
2024-07-15 16:03:24.427190 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.de11.tv
2024-07-15 16:03:24.427571 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.de12.tv
2024-07-15 16:03:24.427600 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.de13.tv
2024-07-15 16:03:24.427704 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.de14.tv
2024-07-15 16:03:24.427740 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.de15.tv
2024-07-15 16:03:24.427774 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.LastScanned.tv
2024-07-15 16:03:24.428180 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.bskyb__tv_.tv
2024-07-15 16:03:24.429163 <Debug> [e2se.gui] e2db::parse_e2db_bouquet  filename: bouquets.radio
2024-07-15 16:03:24.429194 <Error> [e2se.gui] e2db::parse_e2db_bouquet  Parser Error: userbouquet (bouquets.radio:5)
2024-07-15 16:03:24.429221 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.abm.terrestrial_ie_saorview_PSB1.main.radio
2024-07-15 16:03:24.429819 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: userbouquet.de00.radio
2024-07-15 16:03:24.429856 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: ���
2024-07-15 16:03:24.429870 <Debug> [e2se.gui] e2db::parse_e2db_userbouquet  filename: S(    �

Screenshot

No response

harmlessdrudge commented 2 months ago

Sorry, platform: Linux Other DE (Cinnamon); Architecture: X86_64

ctlcltd commented 2 months ago

Thank you

Can you confirm that you are using version 1.5?

Unfortunately the log you provided me is a very small part of the log, if you can upload it in full it would be very helpful.

Those FTP errors are due to the presence of directories (from decoder) in the bouquets directory. To overcome these errors file listing with a more advanced FTP command is used.

It seems that the decoder Set-Top-Box does not have the FTP listing MLSD command, the software fallback to NLST listing. As I do not have the interesting log part, I cannot know.

This command was introduced in the software since version 1.1.1

Please consider that I do not have a ZGemma H7S decoder, I tested on an Edision decoder and the FTP command works.

These FTP errors are - as returned - by the Curl library, used in the software:

FTP Error: Remote file not found [download_data] (/etc/enigma2/AutoBouquetsMaker) FTP Error: Remote file not found [download_data] (/etc/enigma2/crossepg) FTP Error: Remote file not found [download_data] (/etc/tuxbox/config) FTP Error: Remote file not found [download_data] (/etc/tuxbox/scce)

In brief: you can ignore them, those files are not bouquets and will not be downloaded.

I did not want to create a whitelist or blacklist of files to download from the decoder, there may be other settings files that can be opened by the editor in the near future. This point can be re-evaluated, for example: if there are files that are denied access by the decoder.

It would be interesting to understand if your decoder supports the FTP listing MLSD command or if an error of some kind has occurred. The FTP log would be useful. If you want provide it, follow the online help to "Enable FTP debug", Troubleshooting page. If you send the log with FTP debug, please omit the connection parameters to your decoder.

As for parsing errors it could be a bouquet filename error, since version 1.3, the filename should no longer be a problem.

To better investigate the parsing issue it would be useful to have the list you used, if possible.

DVB-S/2 source is Astra 28.2E; DVB-T sources is Saorview

Are they public lists?

Last thing, regarding the picons, currently the software does not download and upload the picons via FTP. Picons are managed by folder in the local filesystem. I am evaluating this point, considering that it is inconvenient and slow to download and load the picons file by file into the decoder, while it is more convenient and fast to upload a Zip file to decompress into the decoder. In the near future there may be an option to separately upload and download picons via FTP.

harmlessdrudge commented 2 months ago

Yes, I am using version 1.5 and apologies for not saying so. Thanks for the background info.

I tried to upload a full log but it was too big (exceeded allowed 64k characters here) so I had to truncate it.

I'll look at an FTP debug tomorrow but in case it's useful in the meantime: the ftp server is vsftpd. Here's the conf file (I don't have time to review right now)


# Example config file /etc/vsftpd.conf
#
# The default compiled in settings are fairly paranoid. This sample file
# loosens things up a bit, to make the ftp daemon more usable.
# Please see vsftpd.conf.5 for all compiled in defaults.
#
# READ THIS: This example file is NOT an exhaustive list of vsftpd options.
# Please read the vsftpd.conf.5 manual page to get a full idea of vsftpd's
# capabilities.

# run standalone
listen=NO
listen_ipv6=YES

# Allow anonymous FTP? (Beware - allowed by default if you comment this out).
anonymous_enable=NO
#
# Uncomment this to allow local users to log in.
local_enable=YES
#
# Uncomment this to enable any form of FTP write command.
write_enable=YES
#
# Default umask for local users is 077. You may wish to change this to 022,
# if your users expect that (022 is used by most other ftpd's)
local_umask=022
#
# Uncomment this to allow the anonymous FTP user to upload files. This only
# has an effect if the above global write enable is activated. Also, you will
# obviously need to create a directory writable by the FTP user.
#anon_upload_enable=YES
#
# Uncomment this if you want the anonymous FTP user to be able to create
# new directories.
#anon_mkdir_write_enable=YES
#
# Activate directory messages - messages given to remote users when they
# go into a certain directory.
dirmessage_enable=YES
#
# Activate logging of uploads/downloads.
xferlog_enable=NO
#
# Make sure PORT transfer connections originate from port 20 (ftp-data).
connect_from_port_20=NO
#
# If you want, you can arrange for uploaded anonymous files to be owned by
# a different user. Note! Using "root" for uploaded files is not
# recommended!
#chown_uploads=YES
#chown_username=whoever
#
# You may override where the log file goes if you like. The default is shown
# below.
#xferlog_file=/var/log/vsftpd.log
#
# If you want, you can have your log file in standard ftpd xferlog format
xferlog_std_format=YES
#
# You may change the default value for timing out an idle session.
#idle_session_timeout=600
#
# You may change the default value for timing out a data connection.
#data_connection_timeout=120
#
# It is recommended that you define on your system a unique user which the
# ftp server can use as a totally isolated and unprivileged user.
#nopriv_user=ftp
#
# Enable this and the server will recognise asynchronous ABOR requests. Not
# recommended for security (the code is non-trivial). Not enabling it,
# however, may confuse older FTP clients.
#async_abor_enable=YES
#
# By default the server will pretend to allow ASCII mode but in fact ignore
# the request. Turn on the below options to have the server actually do ASCII
# mangling on files when in ASCII mode.
# Beware that turning on ascii_download_enable enables malicious remote parties
# to consume your I/O resources, by issuing the command "SIZE /big/file" in
# ASCII mode.
# These ASCII options are split into upload and download because you may wish
# to enable ASCII uploads (to prevent uploaded scripts etc. from breaking),
# without the DoS risk of SIZE and ASCII downloads. ASCII mangling should be
# on the client anyway..
#ascii_upload_enable=YES
#ascii_download_enable=YES
#
# You may fully customise the login banner string:
#ftpd_banner=Welcome to blah FTP service.
#
# You may specify a file of disallowed anonymous e-mail addresses. Apparently
# useful for combatting certain DoS attacks.
#deny_email_enable=YES
# (default follows)
#banned_email_file=/etc/vsftpd.banned_emails
#
# You may specify an explicit list of local users to chroot() to their home
# directory. If chroot_local_user is YES, then this list becomes a list of
# users to NOT chroot().
chroot_list_enable=NO
# (default follows)
#chroot_list_file=/etc/vsftpd.chroot_list
#
# You may activate the "-R" option to the builtin ls. This is disabled by
# default to avoid remote users being able to cause excessive I/O on large
# sites. However, some broken FTP clients such as "ncftp" and "mirror" assume
# the presence of the "-R" option, so there is a strong case for enabling it.
#ls_recurse_enable=YES
#
# This string is the name of the PAM service vsftpd will use.
pam_service_name=vsftpd
#
# This option is examined if userlist_enable is activated. If you set this
# setting to NO, then users will be denied login  unless  they are  explicitly 
# listed  in the file specified by userlist_file.  When login is denied, the 
# denial is issued before the user is asked for a password.
userlist_deny=YES
#
# If enabled, vsftpd will load a list of usernames, from the filename given by
# userlist_file.  If a user tries to log in using  a  name in  this  file,  they
# will be denied before they are asked for a password. This may be useful in 
# preventing cleartext passwords being transmitted. See also userlist_deny.
userlist_enable=NO
#
# If enabled,  vsftpd  will display directory listings with the time in your
# local time zone. The default is to display GMT. The times returned by the
# MDTM FTP command are also affected by this option.
use_localtime=YES
#
# If set to YES, local users will be (by default) placed in a chroot() jail in
# their home directory after login.  Warning: This  option has  security  
# implications,  especially  if  the users have upload permission, or shell access.
# Only enable if you know what you are doing.  Note that these security implications
# are not vsftpd specific. They apply to all FTP daemons which offer to put 
# local  users in chroot() jails.
chroot_local_user=NO
#
allow_writeable_chroot=YES
#
tcp_wrappers=NO

# Enable FXP
pasv_promiscuous=YES
port_promiscuous=YES

# Allow users w/o shell access
check_shell=NO

# Start directory for non-chrooted users
local_root=/
ctlcltd commented 2 months ago

thanks

You already told me version 1.5, I asked you for confirmation and it is OK.

I am sorry, I may have told you something incorrect. I have the same server vsftpd on my test decoder, the difference is that my decoder software does not have sub-directories (AutoBouquetsMaker, etc.) on the bouquets directory.

At the current state of version 1.5, those errors can be ignored, they are soft-errors and do not affect the download of bouquets from the decoder.

I may re-evaluate the issue in future releases and ensure that those directories are excluded in any case. I agree with you that errors can bore users.

The parsing issue is very important.

I found the problem origin: 1:519:1:0:0:0:0:0:0:0:FROM BOUQUET exceeds the size in the parser. Therefore lists type 519 are not currently supported. This parsing issue will be fixed in the next release.

If you want send FTP debug, you can use attachments here on GitHub: https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/attaching-files

Please, if you send FTP debug, omit host address, username and password.

harmlessdrudge commented 2 months ago

e2satdebug.txt

OK, that's one mistake apiece :-)

Debug file attached.

Any idea when the next release will be? Just wondering if you'll do a bugfix release or hold this until you have more things planned. If it's imminent I'll defer cleaning up my bouquets after an update to OpenViX I've just done unless you also assure me that there will be no negative effect on uploading to my system. Sorry I couldn't get to this earlier.

ctlcltd commented 2 months ago

Thank you very much.

The problem has already been fixed in the next coming commit, it is in the works.

I found info about the 519 flag, they are hidden userbouquets. It seems that there is not widespread support for this flag by decoders STB, maybe by some plugins. In any case bouquets should be parsed if there are no big incompatibilities, you have done an excellent issue reporting.

The next release may take some time, in the meantime you can act in different ways:

This thing:

you also assure me that there will be no negative effect on uploading to my system

Side effects on your system are unlikely to occur, it is a use of software with a device. Consider that you are uploading your channel lists to a device via FTP, this may be considered an advanced use of your decoder. I do not know how the manufacturer's warranty on your decoder works.

These lists of favorites are called userbouquets, there is the prefix user- because they are user lists. For example, the most negative effect could be compromised or partial lists. Favorites lists can jump either using software or using the decoder itself or a plugin or whatever. These are things can happen, acceptable side effects.

It is not possible to guarantee anything, also due to the technical nature of loading channel lists to the decoder. It cannot be guaranteed by any software.

In any case it is good practice to make a backup of channel lists before proceeding with changes.

harmlessdrudge commented 2 months ago

E-Channelizer? Hmm. I use Linux and only use one Windows app (MyLifeOrganized under WINE on a miniPC on my desk which acts as part of my local cloud). I've also downloaded and had a quick look at DemonEditor. I preferred the UI of your app and hoped to use it; haven't used any app for this so far beyond the web UI of the box, which uses OpenWebIf as do most Enigma2 STBs.

The box is inexpensive and long past warranty. I can reflash it easily enough if ever necessary and, yes, of course, I can back things up. :-)

I think the simplest solution will be to unhide the hidden bouquets. I suspect they're the Shopping, Children, Special Interest (religion) etc things I've hidden to declutter the screen. I'll check that and confirm just FYI.

ctlcltd commented 2 months ago

It was just an example... I really like DreamboxEdit ❤️ but it runs only on Windows.

Unfortunately it takes a while to release, there are many things to check and check on each release.

I hope to release soon as possible.

harmlessdrudge commented 2 months ago

Sharing this just in case it's useful. Suffice it to say that simply unhiding bouquets marked hidden was not enough.

FYI the hidden (not displayed on screen) bouquets did, initially, appear to be things I marked "hide" -- apart from two which may be intrinsic to how channels are managed or numbered internally. That is, I presume they're not real bouquets at all

userbouquet.abm.terrestrial_ie_saorview_PSB1.separator.tv

#NAME Separator
#SERVICE 1:64:0:0:0:0:0:0:0:0:
#DESCRIPTION Separator
#SERVICE 1:320:0:0:0:0:0:0:0:0:
#SERVICE 1:320:0:0:0:0:0:0:0:0:
#SERVICE 1:320:0:0:0:0:0:0:0:0:
#SERVICE 1:320:0:0:0:0:0:0:0:0:
etc (973 lines)

userbouquet.abm.sat_282_freesat.separator.tv 1 #NAME Separator 2 #SERVICE 1:64:0:0:0:0:0:0:0:0: 3 #DESCRIPTION Separator 4 #SERVICE 1:320:0:0:0:0:0:0:0:0: 5 #SERVICE 1:320:0:0:0:0:0:0:0:0: 6 #SERVICE 1:320:0:0:0:0:0:0:0:0: 7 #SERVICE 1:320:0:0:0:0:0:0:0:0: 8 #SERVICE 1:320:0:0:0:0:0:0:0:0: 9 #SERVICE 1:320:0:0:0:0:0:0:0:0: 10 #SERVICE 1:320:0:0:0:0:0:0:0:0: etc (850 lines)

The bouquets with ".abm" in the name are generated by an Enigma2 extension called autobouquetsmaker (ABM) which runs at a scheduled time, usually overnight, on most boxes with it installed (picks up and handles satellite frequency shifts without user intervention).

Existing hidden bouquets appeared to be zero length files corresponding to the original files (not sure about this). Just unhiding them and then running the download again didn't change anything in terms of download error messages. I deleted all "ABM" bouquets and the crossepg directory on the STB*, regenerated them and ran the download again.

*CrossEPG extension is obsolete as of the upgrade of OpenViX from 6.4 to 6.6 and that will presumably apply to related Enigma2 versions (OpenATV, OpenPLI etc).

INPUT:

#NAME Bouquets (TV)
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.terrestrial_ie_saorview_PSB1.main.tv" ORDER BY bouquet
#SERVICE 1:519:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.terrestrial_ie_saorview_PSB1.separator.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.terrestrial_ie_saorview_PSB1.hd.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.terrestrial_ie_saorview_PSB1.ftahd.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.terrestrial_ie_saorview_PSB1.fta.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.main.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.101.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.200.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.300.tv" ORDER BY bouquet
#SERVICE 1:519:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.400.tv" ORDER BY bouquet
#SERVICE 1:519:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.500.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.600.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.650.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.700.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.800.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.949.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.970.tv" ORDER BY bouquet
#SERVICE 1:519:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.998.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.999.tv" ORDER BY bouquet
#SERVICE 1:519:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.separator.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.hd.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.woodlandstv__tv_.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de00.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de01.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de02.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de03.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de04.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de05.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de06.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de07.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de08.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de09.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de10.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de11.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de12.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de13.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de14.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de15.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.LastScanned.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.bskyb__tv_.tv" ORDER BY bouquet 

OUTPUT:

#NAME Bouquets (TV)
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.terrestrial_ie_saorview_PSB1.main.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.terrestrial_ie_saorview_PSB1.hd.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.terrestrial_ie_saorview_PSB1.ftahd.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.terrestrial_ie_saorview_PSB1.fta.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.main.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.101.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.200.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.300.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.949.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.abm.sat_282_freesat.hd.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.woodlandstv__tv_.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de00.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de01.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de02.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de03.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de04.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de05.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de06.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de07.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de08.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de09.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de10.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de11.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de12.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de13.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de14.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.de15.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.LastScanned.tv" ORDER BY bouquet
#SERVICE 1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.bskyb__tv_.tv" ORDER BY bouquet

The errors remained unchanged apart from the disappearance of that for crossepg.

I've attached 4 screen grabs I took when trying to make sense of things using Beyond Compare

The 400.tv, 500.tv and 998.tv lines correspond to bouquets for "Lifestyle", "Music" and "Freesat 4K" and what they appear to have in common, also with the separator.tv bouquets is that they contain no services.

Screen grabs:

directories.png comparison of saved files and those on STB dotradio and dottv png files: compared with STB files

I ran DemonEditor (which hung for me when I tried it at first) and compared the downloaded bouquets.tv files (png file attached). I'll try this for now but overall I still prefer e2satEditor's simpler user interface. directories dotradio dottv demoneditor_dottv

ctlcltd commented 2 months ago

Thanks

apart from two which may be intrinsic to how channels are managed or numbered internally

Yes, it could be used internally to hide them. In any case the line should be parsed by the software when it has userbouquet file. At the moment I do not know if this flag is used by decoders, plugins, scripts. I quickly searched in some forum posts, I found this issue where they talk about the flag: https://github.com/oe-alliance/AutoBouquetsMaker/issues/74

Just unhiding them and then running the download again didn't change anything in terms of download error messages.

This will be fixed in the next release. The error is due to the fact that listing with NLST command there are directories, this soft error is wanted with NLST, directories cannot be downloaded. Listing with MLSD command, directories are discarded but it is not supported everywhere. It show FTP error but it is negligible, a soft error that does not compromise the download.

All userbouquets with flag other than 7 will not be parsed, there are parser errors and there is no userbouquet filename. This is the main issue.

The errors remained unchanged apart from the disappearance of that for crossepg.

Yes, from version 1.3 to version 1.5, it will not parse userbouquets marked with flag other than 7, it will only parse 1:7:X (X: 1 = TV, 2 = Radio).

As I told you, version 1.2, if you want to try it, does not has this obstacle of the flag but other limitations and imperfections.

Thanks for your time, much appreciated!

I leave the issue open, as soon there is the new release I will let you know.

harmlessdrudge commented 2 months ago

Great, many thanks.

ctlcltd commented 2 months ago

Fixed with commits https://github.com/ctlcltd/e2-sat-editor/commit/5e7ad1e3ec391c49a09e68414b321beec31a7721 https://github.com/ctlcltd/e2-sat-editor/commit/3e5808a13a1f1526b98b878cc2921022db9c44f7 https://github.com/ctlcltd/e2-sat-editor/commit/3e75f6b656b2f39c9005a8d79941ec5dcca4386e https://github.com/ctlcltd/e2-sat-editor/commit/6693d2ac859a494a19a8ee005c3a42df91495f61 Fixed on release v1.6.0

@harmlessdrudge new version 1.6.0 released