Alfresco 6 Support

[deas]

Hi Peter,

I needed this thing working on Alfresco 6 and it looks like I succeeded. As you point out, version in module.properties requires a bump. Other than that, jackson shipping with Alfresco 6 has a slightly different package structure. Had to drop the sdk parent to get proper deps in.

You might remember that I am not a big fan of amp. Hence, I shifted things around a bit to make them more jar friendly.

This is actually the first time I am using this tool and from what I can tell, it still behaves as advertised.

You might want to check my fork.

Just let me know in case you'd like me to provide you with a PR.

regards Andreas

[pmonks]

A PR would be great!

Just out of interest. are the changes backwards compatible with Alfresco 5.x?

[deas]

Package structure of jackson has changed a bit, but I think we can still target all versions of Alfresco with one artifact. It is really about two class imports and constructor initialization we need to replace - namely ObjectMapper and TypeReference. The easiest approach I could come up with is reflection. I'd be fine it and give it a try.

Feel free to test drive my fork before the PR comes in. It's just clone and mvn install.

Offtopic here, but I have just run into another issue. I need cm:content property type support. It is not there yet, is it?

Any opinion about a potential implementation of that?

[pmonks]

Took a look at this this evening, and will continue chipping away at it as I have time.

<mini-rant> It's quite disappointing to see that AMP support has basically been dropped by Alfresco, necessitating the use of ghastly hacks such as the Maven shade plugin. That's going to cause no end of pain for users and customers who deploy more than a small number of extensions into the same JVM... </mini-rant>

Regarding d:content properties, that's tracked as issue #5. I've added more commentary there.

[palojano]

Hi Peter, what is your plan to support Alfresco 6? Regards, Pavel.

[pmonks]

As mentioned above a PR would be welcome.

Sadly the Alfresco v6 SDK broke backwards compatibility of the build, and I have minimal time to spend on fixing things that previously worked.

[andrm]

@pmonks can you point to documentation where AMPs have been dropped? https://docs.alfresco.com/6.1/concepts/dev-modules.html seems to imply that it is still possible.

[pmonks]

@andrm according to https://docs.alfresco.com/6.1/concepts/dev-extensions-packaging-techniques-amps.html, repository AMPs (such as the bulk import tool) are still dropped in the ‘amps’ directory, then the MMT is used to install them (as was the case in earlier versions).

This means the installation instructions should, in theory, remain the same. However there have been several reports (both here and on the mailing list) that this does not work, even after the module’s max version property is updated, suggesting that backwards compatibility has not been preserved.

[apiening]

May I ask if there is an update on this topic or what the current state is? We're using Alfresco 6.1 and need to migrate our data from a 5.1 installation. Is @deas fork still the best bet? Or are there other options to import the files that has been exported with the bulk-export tool?

[pmonks]

@apiening I'm still awaiting a PR (from @deas or whoever has gotten AMP builds working on Alfresco v6). My recollection is that @deas' solution devolved to JAR packaging, but that's not a viable solution long term for a tool such as this.

[apiening]

@pmonks Thank you for the quick update. This does not sound like a short term solution to me.

While searching for a solution I stumbled upon the official Alfresco 6.1 docs at https://docs.alfresco.com/6.1/concepts/bulk-import-via-the-ui.html talking about bulk-import as if it were a core feature. I tried to call the url prefix /alfresco/service/bulkfsimport on my Alfresco 6.1 install and I got a Bulk Filesystem Import Tool form where I can choose the import directory, target NodeRef and initiate the bulk import. I'm confused about that, honestly. I've not yet tried to do an import. Can you tell how this integrated feature compares to this alfresco-bulk-import project?

[deas]

I understand there are people who prefer amps over jars. I am the opposite side of the spectrum. My fork solved a few issues for me. To be honest, I don't want to worry about splitting it apart to deal with one issue at a time or bringing it back to amp.

[shazada]

We're using a jar fork as well and works fine. Haven't tried it with 6.1 do.

[pmonks]

@apiening that's the embedded fork. While you're welcome to use it, keep in mind that it was an unapproved fork of an early pre-release version of the tool, that:

1. hasn't been rebaselined with upstream since it was forked (in 2011 or thereabouts, IIRC)
2. introduced several major bugs / limitations that have never existed in upstream (no support for parallel imports on Alfresco clusters, buggy "striping" algorithm, etc.).

@deas I understand your position, and hope you understand mine. Notably, JAR distribution can't support multi-JAR extensions and although the bulk import tool doesn't depend on any 3rd party JARs right now, there are enhancements being considered that will require that.

[apiening]

Ok got it. Thanks for the the explanations. I gave the embedded version a shot and it failed in the first place due to exported attributes that are not known on the target system. Instead of skipping the unknown attributes it crashed with a stacktrace. I then removed the problematic attributes from the metadata xml files since they are not needed anyways and the import worked finde then.

Is there any kind of manual how to install @deas fork? I would like to give it a try but have not installed alfresco extensions other then AMP based. From my understanding I would need to compile it on my own and need to setup a dev environment just for that. I think the fact that Alfresco 6.1 is based on docker(-compose) doesn't make things easier in that regard.

By accident: Is there an option in alfresco bulk import to ignore specific attributes? That would save me some time writing a script to clean up all metadata xml files by removing these.

[deas]

@pmonks I am perfectly fine with the situation as is.

@apiening You might try downloading a build here. Just drop the jar in alfresco/WEB-INF/lib and it should be good to go.

[pmonks]

@apiening

I gave the embedded version a shot and it failed in the first place due to exported attributes that are not known on the target system. Instead of skipping the unknown attributed it crashed with a stacktrace. I then removed the problematic attributes from the metadata xml files since they are not needed anyways and the import worked finde then.

Yep this is expected behaviour in this case. The tool employs a "fail fast, then pick up where it left off upon rerun" strategy, since that makes it vastly easier to reason about the tool's precise behaviour in the face of a wide variety of failure modes, without imposing the cost of having to redo imports from scratch every time such a failure occurs.

In contrast, silently* ignoring problems is guaranteed to cause unpleasant surprises that are difficult to both triage and fix (undoing an import without resetting your entire repository is almost impossible).

*I consider "silent" to include the behaviour of logging warnings - no one wants to have to trawl through a multi-megabyte log file to find out why their import "succeeded" but didn't do what they expected.

[snip]

By accident: Is there an option in alfresco bulk import to ignore specific attributes?

No there isn't, since that's usually caused by one of two problems, both of which are major and therefore should be flagged for correction, rather than ignored:

1. a custom content model hasn't been deployed to the repository correctly
2. the source XML files have errors

You're welcome to raise an enhancement request for this, of course, but in the interests of transparency I should point out that it's unlikely I'll implement it unless a strong argument can be made in favour of it.

With that said, I had some recent conversations with Symflofy engineers where they pointed out some specific cases (i.e. when a transient network or I/O error occurs) where ignoring the error and automatically retrying does make sense, and their fork of the tool does indeed silently retry in those situations. I'm much more open to that, given that such issues are environmental, and not related to validity problems with the source content set itself.

That would save me some time writing a script to clean up all metadata xml files by removing these.

Alternatively there might be an opportunity to fix whatever code it was that generated the XML files in the first place?

Either way, it's normal to have to write logic that massages the source content set on disk before triggering an import - it's generally easier to deal with those kinds of issues there than after it's been imported into an Alfresco repository...

[pmonks]

@deas

@pmonks I am perfectly fine with the situation as is.

I understand your position, however I'm not sure you appreciate the limitations that that position places on potential future features in the tool.

[deas]

@pmonks Never faced a challenge I could not solve using a jar extension. For me, the worst situation involved dependency shading. That may still not solve all scenarios one could imagine, but it was good enough for me. Same goes for the simplicity of (un-)deployment of jar extensions.

[pmonks]

@deas sure you can require your users to obtain and deploy 3rd party JARs themselves, or build an installer to do it for them, but AMPs already do all of that and more.

Oh and uninstallation of an Alfresco extension is never as simple as removing a file, no matter how your extension is packaged (JAR or AMP). If your extension is anything but code, you're going to inflict a world of pain on the administrator when custom content models etc. are suddenly gone.

[deas]

@pmonks Been blogging about the general lack of extension lifecyle methods years ago. A whole can of worms on its own. No matter how you put it, both approaches have pros and cons. Plain jar works better for me and I never felt the need for amp in my projects. ymmv.

[pmonks]

@deas

Been blogging about the general lack of extension lifecyle methods years ago. A whole can of worms on its own.

Now that's something we violently agree on. 😉

More generally Alfresco simply lacks focus, discipline, or (being blunt) expertise around developer experience. The company has, at times, employed experts in that area, but none of them were given the resources and support to succeed, at least in the ~ decade I worked there. Perhaps this has improved since I left? I haven't seen much evidence to support that argument, but then I'm not working on Alfresco projects day-in day-out as I once was, and most of the folks I'm thinking of have also gone on to bigger & better things.

[deas]

Appears there is more we have in common.

Don't know much about most people who are working at Alfresco today, but I have no doubt they had pretty good engineers back in the days. Also not spending at lot of time with Alfresco these days. Dev experience has always been an issue for me and it appears many (smaller) customers are not yet ready for the container/kubernetes/cloud quantum leap. Some of them are even abandoning it for that reason.

Better things like ... Clojure? Oh wait, actually working on that right now. Right over there. :stuck_out_tongue_winking_eye:

[pmonks]

I have no doubt they had pretty good engineers back in the days

Sure, however most of those same engineers didn't recognise that there's a difference between designing and building a developer experience for themselves vs designing and building a developer experience for 3rd party developers. The ongoing conflation by Alfresco engineering of these two markedly different use cases resulted in the second use case constantly getting short shrift, and was an ongoing source of friction for those employees who were 3rd party developer facing and had to try to explain to 3rd party developers why their experience was so poor. Given that most of those folks have left the company, I see little reason to be optimistic that Alfresco will ever "grok" the need for a robust developer experience in their products. It's just not in the company's DNA.

But on a happier note, welcome to Clojure land! The technology is everything (and more!) that it's cracked up to be! 😉

[deas]

Welcome back actually. After pretty much three years of absence. In fact it is my work in Clojure land that is bringing me back now.

And you know what? This thing is dev tooling. Dev tooling I (and this time another guy at this time) are building for ourselves and others. That is why we care about dev- and user experience. I hope we'll do better than Alfresco. The thing is right over here 😜