search

omeka-s-modules / Osii

Import items from other Omeka S installations
GNU General Public License v3.0
2 stars 0 forks source link

Testing: Import value annotations #5

Closed jimsafley closed 2 years ago

jimsafley commented 2 years ago

The value-annotations branch enables the import of value annotations.

To test this, you'll need to find a remote installation that has value annotations. If you cannot find one—and you likely won't—it's fine to use an installation that you control. Technically, the remote installation and local installation can be one and the same, but I prefer that they are separate installations, because I want property ID mismatches. (If we import from the same installation, we can't test for accurate mapping since property IDs are the same.)

On the remote installation, add quite a few value annotations to multiple values. Make sure to use multiple property/data type combinations, preferably using properties from vocabularies you've imported yourself. Make sure to include "resource" data types as well. Then, on the local installation, add a new import, linking it to the remote installation. On the manage import page, take a snapshot, prepare the import by mapping data types, and import. After this is completed, the newly created items should have values that have value annotations that are identical to the ones on the remote installation.

Because I made some critical changes to the code, you should test this thoroughly, referring back to the original testing issue (#1). You should check for expected behavior for the entire import, not just value annotations. Thanks!

katknow commented 2 years ago

I'm getting this error when trying to import a snapshot:

2022-09-12T19:48:18+00:00 INFO (6): Iterating OSII media: 1666, 1667, 1668, 1669, 1670, 1671, 1672, 1673, 1674, 1675, 1676, 1677, 1678, 1679, 1680, 1681, 1682, 1683, 1684, 1685, 2022-09-12T19:48:18+00:00 INFO (6): Iterating OSII media: 1686, 1687, 1688, 1689, 1690, 1691, 1692, 1693, 1694, 1695, 1696, 1697, 1698, 1699, 1700, 1701, 1702, 1703, 1704, 1705, 2022-09-12T19:48:18+00:00 INFO (6): Iterating OSII media: 1706, 1707, 1708, 1709, 1710, 1711, 1712, 1713, 1714, 1715, 1716, 1717, 1718, 1719, 1720, 1721, 1722, 1723, 1724, 1725, 2022-09-12T19:48:18+00:00 INFO (6): Iterating OSII media: 1726, 1727, 1728, 1729, 1730, 1731, 1732, 1733, 1734, 1735, 1736, 1737, 1738, 1739, 1740, 1741, 1742, 1743, 1744, 1745, 2022-09-12T19:48:18+00:00 INFO (6): Iterating OSII media: 1746, 1747, 1748, 1749, 1750, 1751, 1752, 1753, 1754, 1755, 1756, 1757, 1758, 1759, 1760, 1761, 1762, 1763, 1764, 1765, 2022-09-12T19:48:19+00:00 INFO (6): Iterating OSII media: 1766, 1767, 1768, 1769, 1770, 1771, 1772, 1773, 1774, 1775, 1776, 1777, 1778, 1779, 1780, 1781, 1782, 1783, 1784, 1785, 2022-09-12T19:48:19+00:00 INFO (6): Iterating OSII media: 1786, 1787, 1788, 1789, 1790, 1791, 1792, 1793, 1794, 1795, 1796, 1797, 1798, 1799, 1800, 1801, 1802, 1803, 1804, 1805, 2022-09-12T19:48:19+00:00 INFO (6): Iterating OSII media: 1806, 1807, 1808, 1809, 1810, 1811, 1812, 1813, 1814, 1815, 1816, 1817, 1818, 1819, 1820, 1821, 1822, 1823, 1824, 1825, 2022-09-12T19:48:19+00:00 INFO (6): Iterating OSII media: 1826, 1827, 1828, 1829, 1830, 1831, 1832, 1833, 1834, 1835, 1836, 1837, 1838, 1839, 1840, 1841, 1842, 1843, 1844, 1845, 2022-09-12T19:48:19+00:00 INFO (6): Iterating OSII media: 1846, 1847, 1848, 1849, 1850, 1851, 1852, 1853, 1854, 1855, 1856, 1857, 1858, 1859, 1860, 1861, 1862, 1863, 1864, 1865, 2022-09-12T19:48:19+00:00 INFO (6): Iterating OSII media: 1866, 1867, 1868, 1869, 1870, 1871, 1872, 1873, 1874, 1875, 1876, 1877, 1878, 1879, 1880, 1881, 1882, 1883, 1884, 1885, 2022-09-12T19:48:19+00:00 INFO (6): Iterating OSII media: 1886, 1887, 1888, 1889, 1890, 1891, 1892, 1893, 1894, 1895, 1896, 1897, 1898, 1899, 1900, 1901, 1902, 1903, 1904, 1905, 2022-09-12T19:48:20+00:00 INFO (6): Iterating OSII media: 1906, 1907, 1908, 1909, 1910, 1911, 1912, 1913, 1914, 1915, 1916, 1917, 1918, 1919, 1920, 1921, 1922, 1923, 1924, 1925, 2022-09-12T19:48:20+00:00 INFO (6): Iterating OSII media: 1926, 1927, 1928, 1929, 1930, 1931, 1932, 1933, 1934, 1935, 1936, 1937, 1938, 1939, 1940, 1941, 1942, 1943, 1944, 1945, 2022-09-12T19:48:20+00:00 INFO (6): Iterating OSII media: 1946, 1947, 1948, 1949, 1950, 1951, 1952, 1953, 1954, 1955, 1956, 1957, 1958, 1959, 1960, 1961, 1962, 1963, 1964, 1965, 2022-09-12T19:48:20+00:00 INFO (6): Iterating OSII media: 1966, 1967, 1968, 1969, 1970, 1971, 1972, 1973, 1974, 1975, 1976, 1977, 1978, 1979, 1980, 1981, 1982, 1983, 1984, 1985, 2022-09-12T19:48:20+00:00 INFO (6): Iterating OSII media: 1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2022-09-12T19:48:20+00:00 INFO (6): Iterating OSII media: 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025, 2022-09-12T19:48:20+00:00 INFO (6): Iterating OSII media: 2026, 2027, 2028, 2029, 2030, 2031, 2032, 2033, 2034, 2035, 2036, 2037, 2038, 2039, 2040, 2041, 2042, 2043, 2044, 2045, 2022-09-12T19:48:21+00:00 INFO (6): Iterating OSII media: 2046, 2047, 2048, 2049, 2050, 2051, 2052, 2053, 2054, 2055, 2056, 2057, 2058, 2059, 2060, 2061, 2062, 2063, 2064, 2065, 2022-09-12T19:48:21+00:00 INFO (6): Iterating OSII media: 2066, 2067, 2068, 2069, 2070, 2071, 2072, 2073, 2074, 2075, 2076, 2077, 2078, 2079, 2080, 2081, 2082, 2083, 2084, 2085, 2022-09-12T19:48:21+00:00 INFO (6): Iterating OSII media: 2086, 2087, 2088, 2089, 2090, 2091, 2092, 2093, 2094, 2095, 2096, 2097, 2098, 2099, 2100, 2101, 2102, 2103, 2104, 2105, 2022-09-12T19:48:21+00:00 INFO (6): Iterating OSII media: 2106, 2107, 2108, 2109, 2110, 2111, 2112, 2113, 2114, 2115, 2116, 2117, 2118, 2119, 2120, 2121, 2122, 2123, 2124, 2125, 2022-09-12T19:48:21+00:00 INFO (6): Iterating OSII media: 2126, 2127, 2128, 2129, 2130, 2131, 2132, 2133, 2134, 2135, 2136, 2137, 2138, 2139, 2140, 2141, 2142, 2143, 2144, 2145, 2022-09-12T19:48:21+00:00 INFO (6): Iterating OSII media: 2146, 2147, 2148, 2149, 2150, 2151, 2152, 2153, 2154, 2155, 2156, 2157, 2158, 2159, 2160, 2161, 2162, 2163, 2164, 2165, 2022-09-12T19:48:21+00:00 INFO (6): Iterating OSII media: 2166, 2167, 2168, 2169, 2170, 2171, 2172, 2173, 2174, 2175, 2176, 2177, 2178, 2179, 2180, 2181, 2182, 2183, 2184, 2185, 2022-09-12T19:48:21+00:00 INFO (6): Iterating OSII media: 2186, 2187, 2188, 2022-09-12T19:48:22+00:00 INFO (6): Iterating OSII items: 3239, 3240, 3241, 3242, 3243, 3244, 3245, 3246, 3247, 3248, 3249, 3250, 3251, 3252, 3253, 3254, 3255, 3256, 3257, 3258, 3259, 3260, 3261, 3262, 3263, 3264, 3265, 3266, 3267, 3268, 3269, 3270, 3271, 3272, 3273, 3274, 3275, 3276, 3277, 3278, 3279, 3280, 3281, 3282, 3283, 3284, 3285, 3286, 3287, 3288, 3289, 3290, 3291, 3292, 3293, 3294, 3295, 3296, 3297, 3298, 3299, 3300, 3301, 3302, 3303, 3304, 3305, 3306, 3307, 3308, 3309, 3310, 3311, 3312, 3313, 3314, 3315, 3316, 3317, 3318, 3319, 3320, 3321, 3322, 3323, 3324, 3325, 3326, 3327, 3328, 3329, 3330, 3331, 3332, 3333, 3334, 3335, 3336, 3337, 3338, 2022-09-12T19:48:23+00:00 INFO (6): Iterating OSII items: 3339, 3340, 3341, 3342, 3343, 3344, 3345, 3346, 3347, 3348, 3349, 3350, 3351, 3352, 3353, 3354, 3355, 3356, 3357, 3358, 3359, 3360, 3361, 3362, 3363, 3364, 3365, 3366, 3367, 3368, 3369, 3370, 3371, 3372, 3373, 3374, 3375, 3376, 3377, 3378, 3379, 3380, 3381, 3382, 3383, 3384, 3385, 3386, 3387, 3388, 3389, 3390, 3391, 3392, 3393, 3394, 3395, 3396, 3397, 3398, 3399, 3400, 3401, 3402, 3403, 3404, 3405, 3406, 3407, 3408, 3409, 3410, 3411, 3412, 3413, 3414, 3415, 3416, 3417, 3418, 3419, 3420, 3421, 3422, 3423, 3424, 3425, 3426, 3427, 3428, 3429, 3430, 3431, 3432, 3433, 3434, 3435, 3436, 3437, 3438, 2022-09-12T19:48:24+00:00 ERR (3): PDOException: SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (katknow.value, CONSTRAINT FK_1D775834549213EC FOREIGN KEY (property_id) REFERENCES property (id) ON DELETE CASCADE) in /var/www/html/katknow/OmekaSTesting/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php:112 Stack trace:

0 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php(112): PDOStatement->execute(NULL)

1 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php(177): Doctrine\DBAL\Driver\PDOStatement->execute(NULL)

2 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/Entity/BasicEntityPersister.php(287): Doctrine\DBAL\Statement->execute()

3 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(1122): Doctrine\ORM\Persisters\Entity\BasicEntityPersister->executeInserts()

4 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(420): Doctrine\ORM\UnitOfWork->executeInserts(Object(Doctrine\ORM\Mapping\ClassMetadata))

5 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(376): Doctrine\ORM\UnitOfWork->commit(NULL)

6 /var/www/html/katknow/OmekaSTesting/modules/Osii/src/Job/AbstractOsiiJob.php(173): Doctrine\ORM\EntityManager->flush()

7 /var/www/html/katknow/OmekaSTesting/modules/Osii/src/Job/DoImport.php(300): Osii\Job\AbstractOsiiJob->flushClear()

8 /var/www/html/katknow/OmekaSTesting/application/src/Job/DispatchStrategy/Synchronous.php(34): Osii\Job\DoImport->perform()

9 /var/www/html/katknow/OmekaSTesting/application/src/Job/Dispatcher.php(105): Omeka\Job\DispatchStrategy\Synchronous->send(Object(Omeka\Entity\Job))

10 /var/www/html/katknow/OmekaSTesting/application/data/scripts/perform-job.php(66): Omeka\Job\Dispatcher->send(Object(Omeka\Entity\Job), Object(Omeka\Job\DispatchStrategy\Synchronous))

11 {main}

Next Doctrine\DBAL\Driver\PDO\Exception: SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (katknow.value, CONSTRAINT FK_1D775834549213EC FOREIGN KEY (property_id) REFERENCES property (id) ON DELETE CASCADE) in /var/www/html/katknow/OmekaSTesting/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDO/Exception.php:18 Stack trace:

0 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php(114): Doctrine\DBAL\Driver\PDO\Exception::new(Object(PDOException))

1 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php(177): Doctrine\DBAL\Driver\PDOStatement->execute(NULL)

2 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/Entity/BasicEntityPersister.php(287): Doctrine\DBAL\Statement->execute()

3 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(1122): Doctrine\ORM\Persisters\Entity\BasicEntityPersister->executeInserts()

4 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(420): Doctrine\ORM\UnitOfWork->executeInserts(Object(Doctrine\ORM\Mapping\ClassMetadata))

5 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(376): Doctrine\ORM\UnitOfWork->commit(NULL)

6 /var/www/html/katknow/OmekaSTesting/modules/Osii/src/Job/AbstractOsiiJob.php(173): Doctrine\ORM\EntityManager->flush()

7 /var/www/html/katknow/OmekaSTesting/modules/Osii/src/Job/DoImport.php(300): Osii\Job\AbstractOsiiJob->flushClear()

8 /var/www/html/katknow/OmekaSTesting/application/src/Job/DispatchStrategy/Synchronous.php(34): Osii\Job\DoImport->perform()

9 /var/www/html/katknow/OmekaSTesting/application/src/Job/Dispatcher.php(105): Omeka\Job\DispatchStrategy\Synchronous->send(Object(Omeka\Entity\Job))

10 /var/www/html/katknow/OmekaSTesting/application/data/scripts/perform-job.php(66): Omeka\Job\Dispatcher->send(Object(Omeka\Entity\Job), Object(Omeka\Job\DispatchStrategy\Synchronous))

11 {main}

Next Doctrine\DBAL\Exception\ForeignKeyConstraintViolationException: An exception occurred while executing 'INSERT INTO value (type, lang, value, uri, is_public, resource_id, property_id, value_resource_id, value_annotation_id) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)' with params ["literal", null, "this is a title test", null, 1, 34195, 1568, null, null]:

SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (katknow.value, CONSTRAINT FK_1D775834549213EC FOREIGN KEY (property_id) REFERENCES property (id) ON DELETE CASCADE) in /var/www/html/katknow/OmekaSTesting/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/AbstractMySQLDriver.php:68 Stack trace:

0 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/dbal/lib/Doctrine/DBAL/DBALException.php(182): Doctrine\DBAL\Driver\AbstractMySQLDriver->convertException('An exception oc...', Object(Doctrine\DBAL\Driver\PDO\Exception))

1 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/dbal/lib/Doctrine/DBAL/DBALException.php(159): Doctrine\DBAL\DBALException::wrapException(Object(Doctrine\DBAL\Driver\PDO\MySQL\Driver), Object(Doctrine\DBAL\Driver\PDO\Exception), 'An exception oc...')

2 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(2214): Doctrine\DBAL\DBALException::driverExceptionDuringQuery(Object(Doctrine\DBAL\Driver\PDO\MySQL\Driver), Object(Doctrine\DBAL\Driver\PDO\Exception), 'INSERT INTO `va...', Array)

3 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php(183): Doctrine\DBAL\Connection->handleExceptionDuringQuery(Object(Doctrine\DBAL\Driver\PDO\Exception), 'INSERT INTO `va...', Array, Array)

4 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/Entity/BasicEntityPersister.php(287): Doctrine\DBAL\Statement->execute()

5 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(1122): Doctrine\ORM\Persisters\Entity\BasicEntityPersister->executeInserts()

6 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(420): Doctrine\ORM\UnitOfWork->executeInserts(Object(Doctrine\ORM\Mapping\ClassMetadata))

7 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(376): Doctrine\ORM\UnitOfWork->commit(NULL)

8 /var/www/html/katknow/OmekaSTesting/modules/Osii/src/Job/AbstractOsiiJob.php(173): Doctrine\ORM\EntityManager->flush()

9 /var/www/html/katknow/OmekaSTesting/modules/Osii/src/Job/DoImport.php(300): Osii\Job\AbstractOsiiJob->flushClear()

10 /var/www/html/katknow/OmekaSTesting/application/src/Job/DispatchStrategy/Synchronous.php(34): Osii\Job\DoImport->perform()

11 /var/www/html/katknow/OmekaSTesting/application/src/Job/Dispatcher.php(105): Omeka\Job\DispatchStrategy\Synchronous->send(Object(Omeka\Entity\Job))

12 /var/www/html/katknow/OmekaSTesting/application/data/scripts/perform-job.php(66): Omeka\Job\Dispatcher->send(Object(Omeka\Entity\Job), Object(Omeka\Job\DispatchStrategy\Synchronous))

13 {main}

jimsafley commented 2 years ago

That's odd. Did you happen to delete a vocabulary while the import process was running? Try running the import again to see if it happens regularly.

jimsafley commented 2 years ago

Also, to make sure that this error is unconnected with the new feature, try checking out the main branch and running the import. Are there any errors?

katknow commented 2 years ago

So there weren't errors on main, and when I switched back and tried again on the value annotation branch it did work. It imported everything correctly, including multiple kinds of annotations with vocabularies I had imported myself on different kinds of resources. I did end up importing from my own Omeka S instance since I was having trouble finding annotations, but it was a separate one I had created. However, there is some weirdness going on with how the item page looks:

Screen Shot 2022-09-12 at 5 05 07 PM

I'm guessing this doesn't have anything to do with this, but since it happened while I was trying to sort out this problem I thought I'd mention it.

I'm going to revisit the original testing issue to generally check and make sure all the different elements are working as anticipated.

katknow commented 2 years ago

Other than the weird layout I think this is working okay. Tried with queries, with and without media, with and without item sets, etc.

kimisgold commented 2 years ago

Other than the weird layout I think this is working okay. Tried with queries, with and without media, with and without item sets, etc.

There's been a recent update to metadata styles and markup. Try a hard refresh and let me know if these issues persist.

jimsafley commented 2 years ago

@katknow Do you plan on doing further testing?

katknow commented 2 years ago

@kimisgold yes hard refresh fixed it, thanks.

@jimsafley I played with this a couple more hours last night and was unable to replicate the error so I think we're it may be good to go. I redid the original testing issue as well and things worked as expected. The only caveat is I imported from a second instance I created myself, so everything was purely based on what I could think up myself in terms of annotations, vocabularies etc.

jimsafley commented 2 years ago

Thanks. It's a strange and unexpected error, which gives me pause. Anyway, as usual, I appreciate your thoroughness!