Scanning (3.0)

[HTsybenko]

1. Create a new project
2. Add files to the project (tried files from the user journey project:hap, lext, md, txt, h5, gwy)
3. Click scan (all extractors were updated to the latest versions)

Errors in the terminal: Traceback (most recent call last): File "/home/htsybenko/pasta_3_10/lib/python3.10/site-packages/pasta_eln/guiStyle.py", line 57, in self.clicked.connect(lambda: widget.execute(command)) # type: ignore[attr-defined] File "/home/htsybenko/pasta_3_10/lib/python3.10/site-packages/pasta_eln/GUI/sidebar.py", line 183, in execute self.comm.backend.scanProject(self.progress, self.openProjectId, '') File "/home/htsybenko/pasta_3_10/lib/python3.10/site-packages/pasta_eln/backend.py", line 379, in scanProject self.addData('', {'name':path}, hierStack) File "/home/htsybenko/pasta_3_10/lib/python3.10/site-packages/pasta_eln/backend.py", line 207, in addData self.useExtractors(path,shasum,doc) #create image/content File "/home/htsybenko/pasta_3_10/lib/python3.10/site-packages/pasta_eln/backend.py", line 467, in useExtractors if doc['style']['main'].startswith(doc['type'][0]): KeyError: 'style'

Traceback (most recent call last): File "/home/htsybenko/pasta_3_10/lib/python3.10/site-packages/pasta_eln/GUI/project.py", line 150, in change self.mainL.itemAt(i).widget().setParent(None) File "/home/htsybenko/pasta_3_10/lib/python3.10/site-packages/pasta_eln/GUI/projectLeafRenderer.py", line 165, in sizeHint doc = self.comm.backend.db.getDoc(docID) File "/home/htsybenko/pasta_3_10/lib/python3.10/site-packages/pasta_eln/sqlite.py", line 177, in getDoc doc = dict(cursor.fetchone()) TypeError: 'NoneType' object is not iterable

Verify database leads to the following kinds of messages: dch05: child-number and dirName dont match --12a03dbb73094f1b8e3220d459110b25 (I guess for those files which were successfully imported)

File on harddisk but not DB (2): Test/001_IndentationRaw/Al_20mN_load20s_hold5s.hap

bch01: These files of database not on filesystem(3):

• Test/Aluminum project annotations.txt

[HTsybenko]

Same error occurs also for the "Start analyze and repair", once it occured during scanning: Traceback (most recent call last): File "/home/htsybenko/pasta_3_10/lib/python3.10/site-packages/pasta_eln/guiStyle.py", line 57, in self.clicked.connect(lambda: widget.execute(command)) # type: ignore[attr-defined] File "/home/htsybenko/pasta_3_10/lib/python3.10/site-packages/pasta_eln/GUI/sidebar.py", line 183, in execute self.comm.backend.scanProject(self.progress, self.openProjectId, '') File "/home/htsybenko/pasta_3_10/lib/python3.10/site-packages/pasta_eln/backend.py", line 379, in scanProject self.addData('', {'name':path}, hierStack) File "/home/htsybenko/pasta_3_10/lib/python3.10/site-packages/pasta_eln/backend.py", line 207, in addData self.useExtractors(path,shasum,doc) #create image/content File "/home/htsybenko/pasta_3_10/lib/python3.10/site-packages/pasta_eln/backend.py", line 467, in useExtractors if doc['style']['main'].startswith(doc['type'][0]): KeyError: 'style'

[SteffenBrinckmann]

1. Did you use the old extractors? If yes, extractors use a new system now.
2. If not, can you also attach the log file such that Ii can identify which file, which extractor, caused the issue?

[HTsybenko]

1. Before scanning, the extractors were updated to the latest versions available on the IEK-2 repo.

2. I have repeated the steps today to save the log file: log.md

However, this time (after restarting the PC) the errors mentioned in the initial report were not observed in the terminal (some errors are still present in the log file). Also, after scanning the "verify database" still gives "dch05: child-number and dirName dont match --c20cee88652242ee83fea53e73596064"

3. "Start analyze and repair again" (log also present in the log.md) now has issues with simple.png Traceback (most recent call last): File "/home/htsybenko/pasta_3_10/lib/python3.10/site-packages/pasta_eln/guiStyle.py", line 57, in self.clicked.connect(lambda: widget.execute(command)) # type: ignore[attr-defined] File "/home/htsybenko/pasta_3_10/lib/python3.10/site-packages/pasta_eln/GUI/configSetup.py", line 98, in execute exampleData(True, self.callbackProgress) File "/home/htsybenko/pasta_3_10/lib/python3.10/site-packages/pasta_eln/installationTools.py", line 274, in exampleData docID = df[df['name']=='simple.png']['id'].values[0] IndexError: index 0 is out of bounds for axis 0 with size 0

The "Verify database" shows the following messages

:disk(1) and db(2) content do not match*: x-88b8eb5ef1d7460795deb3e1c8700312 branches differ [{'stack': ['x-34a16b5a7efc40df87c49f17ffb5068a'], 'child': 2, 'path': 'PastasExampleProject/002_DataFiles'}] [{'stack': ['x-77c1509f3d074c53ae6cfa841110889e'], 'child': 2, 'path': 'PastasExampleProject/002_DataFiles'}] key:id --- disk +++ database @@ -1 +1 @@ -x-0d11a42477aa41d9951f13f6071481f5 +x-88b8eb5ef1d7460795deb3e1c8700312key:dateCreated --- disk +++ database @@ -1 +1 @@ -2024-09-09T08:41:59.441560 +2024-09-09T08:31:29.425828key:dateModified --- disk +++ database @@ -1 +1 @@ -2024-09-09T08:41:59.441566 +2024-09-09T08:31:29.425833

[SteffenBrinckmann]

@HTsybenko can you retry from clean restart: I made some changes but cannot understand why the latest issue happened: create example data, create example data, verify Only explanation I have: after 2nd example data, you close the window with ESC and prevent the restart. Then the verify will still think it is the old data and fail. But if you restart, then everything should be great.

[SteffenBrinckmann]

Note for myself: created test in testsComplicated/01 that does exactly this test

[HTsybenko]

@HTsybenko can you retry from clean restart: I made some changes but cannot understand why the latest issue happened: create example data, create example data, verify Only explanation I have: after 2nd example data, you close the window with ESC and prevent the restart. Then the verify will still think it is the old data and fail. But if you restart, then everything should be great.

@SteffenBrinckmann I reinstalled the software again, and the configuration procedure worked without an issue.

The GUI opens without a pre-selected theme now:

After adding files to the new project and scanning it, "Verify database" responds with

Here are the outputs during the scanning:

extractor_md.py /home/htsybenko/Data/New/Indentation procedure.md extractor_txt.py /home/htsybenko/Data/New/Aluminum project annotations.txt Warning, issue with extractor extractor_txt.py extractor_hdf5.py /home/htsybenko/Data/New/Al_10mN_load20s_hold5s.hdf5 Warning, issue with extractor extractor_hdf5.py extractor_gwy.py /home/htsybenko/Data/New/Al_10mN_1.gwy Warning, issue with extractor extractor_gwy.py extractor_hap.py /home/htsybenko/Data/New/Al_10mN_load20s_hold5s.hap python3: can't open file '/home/htsybenko/converter_hap.py': [Errno 2] No such file or directory Warning, issue with extractor extractor_hap.py

And the log:

09-27 09:04:32|DEBUG:Add/edit data in cwd:/home/htsybenko/Data with stack:[] and name: new and type: ['x0'] and edit: False

09-27 09:05:46|DEBUG:project:changeProject |x-19ee08badbb44fd88befeb94aabf7ae8|| 09-27 09:05:46|DEBUG:ProjectView elements at 1: 0 09-27 09:05:46|DEBUG:ProjectView elements at 2: 0 09-27 09:05:46|DEBUG:ProjectView elements at 4: 3 09-27 09:05:47|INFO:Scan: add file to DB: New/Indentation procedure.md 09-27 09:05:47|DEBUG:Add/edit data in cwd:New with stack:['x-19ee08badbb44fd88befeb94aabf7ae8'] and name: New/Indentation procedure.md and type: [''] and edit: False 09-27 09:05:47|INFO:Scan: add file to DB: New/Aluminum project annotations.txt 09-27 09:05:47|DEBUG:Add/edit data in cwd:New with stack:['x-19ee08badbb44fd88befeb94aabf7ae8'] and name: New/Aluminum project annotations.txt and type: [''] and edit: False 09-27 09:05:47|ERROR:ERROR with extractor extractor_txt.py Traceback (most recent call last): File "/home/htsybenko/pasta_3_10/lib/python3.10/site-packages/pasta_eln/backend.py", line 440, in useExtractors module = importlib.import_module(pyFile[:-3]) File "/usr/lib/python3.10/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 1006, in _find_and_load_unlocked File "", line 688, in _load_unlocked File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "/home/htsybenko/pasta_3_10/lib/python3.10/site-packages/pasta_eln/Extractors/extractor_txt.py", line 15, in from diffpy.structure import Atom, Lattice, Structure ModuleNotFoundError: No module named 'diffpy'

09-27 09:05:47|INFO:Scan: add file to DB: New/Al_10mN_load20s_hold5s.hdf5 09-27 09:05:47|DEBUG:Add/edit data in cwd:New with stack:['x-19ee08badbb44fd88befeb94aabf7ae8'] and name: New/Al_10mN_load20s_hold5s.hdf5 and type: [''] and edit: False 09-27 09:05:47|DEBUG:Creating converter from 7 to 5 09-27 09:05:47|DEBUG:Creating converter from 5 to 7 09-27 09:05:47|DEBUG:Creating converter from 7 to 5 09-27 09:05:47|DEBUG:Creating converter from 5 to 7 09-27 09:05:47|DEBUG:Creating converter from 3 to 5 09-27 09:05:47|ERROR:ERROR with extractor extractor_hdf5.py Traceback (most recent call last): File "/home/htsybenko/pasta_3_10/lib/python3.10/site-packages/pastaeln/backend.py", line 451, in useExtractors = json.dumps(doc[meta][item]) File "/usr/lib/python3.10/json/init.py", line 231, in dumps return _default_encoder.encode(obj) File "/usr/lib/python3.10/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/usr/lib/python3.10/json/encoder.py", line 179, in default raise TypeError(f'Object of type {o.class.name} ' TypeError: Object of type ndarray is not JSON serializable

09-27 09:05:47|INFO:Scan: add file to DB: New/Al_10mN_1.gwy 09-27 09:05:47|DEBUG:Add/edit data in cwd:New with stack:['x-19ee08badbb44fd88befeb94aabf7ae8'] and name: New/Al_10mN_1.gwy and type: [''] and edit: False 09-27 09:05:48|ERROR:ERROR with extractor extractor_gwy.py Traceback (most recent call last): File "/home/htsybenko/pasta_3_10/lib/python3.10/site-packages/pasta_eln/backend.py", line 460, in useExtractors if doc['style']['main'].startswith(doc['type'][0]): KeyError: 'style'

09-27 09:05:48|INFO:Scan: add file to DB: New/10_1.lext 09-27 09:05:48|DEBUG:Add/edit data in cwd:New with stack:['x-19ee08badbb44fd88befeb94aabf7ae8'] and name: New/10_1.lext and type: [''] and edit: False 09-27 09:05:48|INFO:Scan: add file to DB: New/Al_10mN_load20s_hold5s.hap 09-27 09:05:48|DEBUG:Add/edit data in cwd:New with stack:['x-19ee08badbb44fd88befeb94aabf7ae8'] and name: New/Al_10mN_load20s_hold5s.hap and type: [''] and edit: False 09-27 09:05:48|ERROR:ERROR with extractor extractor_hap.py Traceback (most recent call last): File "/home/htsybenko/pasta_3_10/lib/python3.10/site-packages/pasta_eln/backend.py", line 460, in useExtractors if doc['style']['main'].startswith(doc['type'][0]): KeyError: 'style'

09-27 09:05:48|INFO:Scan: these files are on DB but not hard disk

09-27 09:05:48|INFO:Scan: these directories are on DB but not hard disk

09-27 09:05:48|INFO:Scan: file already in DB: New/Indentation procedure.md 09-27 09:05:48|INFO:Scan: file already in DB: New/Aluminum project annotations.txt 09-27 09:05:48|INFO:Scan: file already in DB: New/Al_10mN_load20s_hold5s.hdf5 09-27 09:05:48|INFO:Scan: file already in DB: New/Al_10mN_1.gwy 09-27 09:05:48|INFO:Scan: file already in DB: New/10_1.lext 09-27 09:05:48|INFO:Scan: file already in DB: New/Al_10mN_load20s_hold5s.hap 09-27 09:05:48|INFO:Scan: these files are on DB but not hard disk

09-27 09:05:48|INFO:Scan: these directories are on DB but not hard disk

09-27 09:05:48|DEBUG:project:changeProject |x-19ee08badbb44fd88befeb94aabf7ae8|| 09-27 09:05:48|DEBUG:ProjectView elements at 1: 0 09-27 09:05:48|DEBUG:ProjectView elements at 2: 0 09-27 09:05:48|DEBUG:ProjectView elements at 4: 3

[SteffenBrinckmann]

Fixed in da9ddfd3f9199f8b67ddcc2794d379480819a56d